[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:29 dist-test-slave-09fp krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:29 dist-test-slave-09fp krb5kdc[3421](info): set up 2 sockets
Jun 19 18:40:29 dist-test-slave-09fp krb5kdc[3421](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 18:40:30.815161 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:30 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:30.999472 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40001
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:40001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:31.354907 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:31.355289 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:31.355464 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:31.363384 3434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:31.363560 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:31.363680 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:31.363782 3434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:31.363869 3434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:31.373976 3434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:40001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:31.375468 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:31.376879 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:31.390290 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:31.390307 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:31.390348 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:31.390683 3434 server_base.cc:1034] running on GCE node
I20260619 18:40:31.391861 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:31.393436 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:31.394690 3434 hybrid_clock.cc:648] HybridClock initialized: now 1781894431394642 us; error 45 us; skew 500 ppm
I20260619 18:40:31.395074 3434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:31 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894431, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:31.398838 3434 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:31.400446 3434 webserver.cc:460] Webserver started at http://127.3.85.190:42779/ using document root <none> and password file <none>
I20260619 18:40:31.401223 3434 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:31.401381 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:31.401708 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:31.405019 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "1ee4444a4e00494298a2a374f52a2dcd"
format_stamp: "Formatted at 2026-06-19 18:40:31 on dist-test-slave-09fp"
server_key: "935d6aa493479d58a91a8e4ee01acda6"
server_key_iv: "6654d5c2bccbedcbb6bbac03f69dca62"
server_key_version: "encryptionkey@0"
I20260619 18:40:31.405890 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "1ee4444a4e00494298a2a374f52a2dcd"
format_stamp: "Formatted at 2026-06-19 18:40:31 on dist-test-slave-09fp"
server_key: "935d6aa493479d58a91a8e4ee01acda6"
server_key_iv: "6654d5c2bccbedcbb6bbac03f69dca62"
server_key_version: "encryptionkey@0"
I20260619 18:40:31.411875 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:40:31.415954 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:31.417567 3434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:31.417793 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "1ee4444a4e00494298a2a374f52a2dcd"
format_stamp: "Formatted at 2026-06-19 18:40:31 on dist-test-slave-09fp"
server_key: "935d6aa493479d58a91a8e4ee01acda6"
server_key_iv: "6654d5c2bccbedcbb6bbac03f69dca62"
server_key_version: "encryptionkey@0"
I20260619 18:40:31.418030 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:31.446770 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:31.451400 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:31.451722 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:31.472332 3434 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:40001
I20260619 18:40:31.472353 3500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:40001 every 8 connection(s)
I20260619 18:40:31.473891 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:31.478657 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:31.479041 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 3434
I20260619 18:40:31.479368 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:31.481945 3414 external_mini_cluster.cc:1428] Setting key b977408eb96db7728330a464ca30e78c
I20260619 18:40:31.494614 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd: Bootstrap starting.
I20260619 18:40:31.499478 3501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:31.500910 3501 log.cc:824] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:31.504608 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd: No bootstrap required, opened a new log
I20260619 18:40:31.515573 3501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } }
I20260619 18:40:31.516158 3501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:31.516294 3501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee4444a4e00494298a2a374f52a2dcd, State: Initialized, Role: FOLLOWER
I20260619 18:40:31.517073 3501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } }
I20260619 18:40:31.517395 3501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:31.517570 3501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:31.517778 3501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 0 FOLLOWER]: Advancing to term 1
Jun 19 18:40:31 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:31.521626 3501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } }
I20260619 18:40:31.522344 3501 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ee4444a4e00494298a2a374f52a2dcd; no voters:
I20260619 18:40:31.523288 3501 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:31.523419 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:31.524866 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [term 1 LEADER]: Becoming Leader. State: Replica: 1ee4444a4e00494298a2a374f52a2dcd, State: Running, Role: LEADER
I20260619 18:40:31.525580 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } }
I20260619 18:40:31.526207 3501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:31.528967 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee4444a4e00494298a2a374f52a2dcd. Latest consensus state: current_term: 1 leader_uuid: "1ee4444a4e00494298a2a374f52a2dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } } }
I20260619 18:40:31.529476 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:31.530176 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ee4444a4e00494298a2a374f52a2dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ee4444a4e00494298a2a374f52a2dcd" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40001 } } }
I20260619 18:40:31.530663 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:31.537210 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:31.487045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.3.85.190:40001)
0619 18:40:31.488091 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:31.488113 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:31.489826 (+ 1713us) server_negotiation.cc:407] Connection header received
0619 18:40:31.503561 (+ 13735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:31.503595 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:31.504054 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:31.504434 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:31.509314 (+ 4880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:31.510895 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:31.513716 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:31.514672 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:31.514853 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:31.523416 (+ 8563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:31.523439 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:31.523463 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:31.523531 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:31.527848 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:31.529179 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:31.529186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:31.529194 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:31.529440 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:31.530127 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:31.530134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:31.530139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:31.531831 (+ 1692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:31.532272 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:31.535261 (+ 2989us) server_negotiation.cc:299] Negotiation successful
0619 18:40:31.535800 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":433,"threads_started":1}
I20260619 18:40:31.540524 3511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:31.547190 3511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:31.559762 3511 catalog_manager.cc:1347] Generated new cluster ID: c937937fdacf4a94a199cbd5fb4ef44e
I20260619 18:40:31.559937 3511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:31.571213 3511 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:31.572652 3511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:31.598143 3511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee4444a4e00494298a2a374f52a2dcd: Generated new TSK 0
I20260619 18:40:31.599359 3511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:40:31.650271 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.017099 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.017423 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.017530 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.025640 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.025779 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.025904 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:40:32.036345 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:32.037914 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.039366 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.054430 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.054366 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.054407 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.055226 3528 server_base.cc:1034] running on GCE node
I20260619 18:40:32.055938 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.056905 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.058132 3528 hybrid_clock.cc:648] HybridClock initialized: now 1781894432058073 us; error 37 us; skew 500 ppm
I20260619 18:40:32.058553 3528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:32 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.061978 3528 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:40:32.063687 3528 webserver.cc:460] Webserver started at http://127.3.85.129:44865/ using document root <none> and password file <none>
I20260619 18:40:32.064456 3528 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.064602 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.064951 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.068161 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "ee90eaab32f34d2ba43bc1ec84e3547f"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "0b5af7ceae6bc427c4aab1284a9fcf9f"
server_key_iv: "662af4cf7d4045b43f8557b1a462cec0"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.069465 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "ee90eaab32f34d2ba43bc1ec84e3547f"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "0b5af7ceae6bc427c4aab1284a9fcf9f"
server_key_iv: "662af4cf7d4045b43f8557b1a462cec0"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.075822 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:40:32.079946 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.081599 3528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:32.081835 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "ee90eaab32f34d2ba43bc1ec84e3547f"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "0b5af7ceae6bc427c4aab1284a9fcf9f"
server_key_iv: "662af4cf7d4045b43f8557b1a462cec0"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.082079 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.109268 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.114153 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.114568 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.115624 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:32.117077 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:32.117619 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.117786 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:32.117861 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.156973 3528 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:45889
I20260619 18:40:32.157019 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:45889 every 8 connection(s)
I20260619 18:40:32.158349 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:32.160120 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 3528
I20260619 18:40:32.160368 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:32.161160 3414 external_mini_cluster.cc:1428] Setting key 2170dde48441ee0dee809b0260b5e5b5
Jun 19 18:40:32 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:40:32.184235 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.161328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:51495 (local address 127.3.85.190:40001)
0619 18:40:32.161848 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.161855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.165351 (+ 3496us) server_negotiation.cc:407] Connection header received
0619 18:40:32.165411 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.165415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.165498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.165587 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.168173 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.169290 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.171186 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.172454 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.172641 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.177007 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.177016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.177024 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.177072 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.180516 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.181541 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.181546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.181549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.181681 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.182179 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.182183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.182185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.182618 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.182778 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.183777 (+ 999us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.183970 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260619 18:40:32.186131 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.162070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40001 (local address 127.3.85.129:51495)
0619 18:40:32.163103 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.163159 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.164993 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.165983 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.165999 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.166927 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:32.167929 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.167955 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.169505 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.169510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.170990 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.171011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.172759 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.172765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.173028 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.173862 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:32.173887 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:32.176669 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:32.180789 (+ 4120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.180804 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.180831 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.181375 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.181802 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.181805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.181808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.182022 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.182783 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:32.182794 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:32.183325 (+ 531us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.183673 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.184398 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":418,"threads_started":1}
I20260619 18:40:32.188122 3656 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40001
I20260619 18:40:32.188498 3656 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:32.189214 3656 heartbeater.cc:511] Master 127.3.85.190:40001 requested a full tablet report, sending...
I20260619 18:40:32.191529 3466 ts_manager.cc:194] Registered new tserver with Master: ee90eaab32f34d2ba43bc1ec84e3547f (127.3.85.129:45889)
I20260619 18:40:32.193014 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:51495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:40:32.207054 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.568009 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.568325 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.568486 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.576520 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.576663 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.576854 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:40:32.587564 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:32.588949 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.590385 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.604749 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.604707 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.605727 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.606209 3662 server_base.cc:1034] running on GCE node
I20260619 18:40:32.606988 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.608067 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.609333 3662 hybrid_clock.cc:648] HybridClock initialized: now 1781894432609264 us; error 72 us; skew 500 ppm
I20260619 18:40:32.609704 3662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:32 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.613617 3662 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:40:32.615420 3662 webserver.cc:460] Webserver started at http://127.3.85.130:41473/ using document root <none> and password file <none>
I20260619 18:40:32.616194 3662 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.616353 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.616689 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.619916 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "5747a50c7cbf430c9ec5dbf65921e0f1"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "f9461f66ee4bec24bac8521e4a5bf888"
server_key_iv: "d0c91c905df5271588fa490cbd4829fe"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.620769 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "5747a50c7cbf430c9ec5dbf65921e0f1"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "f9461f66ee4bec24bac8521e4a5bf888"
server_key_iv: "d0c91c905df5271588fa490cbd4829fe"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.626698 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 18:40:32.631002 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.632587 3662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:40:32.632839 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "5747a50c7cbf430c9ec5dbf65921e0f1"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-09fp"
server_key: "f9461f66ee4bec24bac8521e4a5bf888"
server_key_iv: "d0c91c905df5271588fa490cbd4829fe"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.633087 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.658411 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.663156 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.663484 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.664611 3662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:32.666097 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:32.666213 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.666376 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:32.666450 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.706426 3662 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40929
I20260619 18:40:32.706641 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40929 every 8 connection(s)
I20260619 18:40:32.708642 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:32.716447 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 3662
I20260619 18:40:32.716679 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:32.717114 3414 external_mini_cluster.cc:1428] Setting key d36c354cc461c60e90e278346071d2a2
Jun 19 18:40:32 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:32.732962 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.712037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:33637 (local address 127.3.85.190:40001)
0619 18:40:32.713395 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.713404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.713955 (+ 551us) server_negotiation.cc:407] Connection header received
0619 18:40:32.716486 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.716492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.716597 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.716708 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.718686 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.719571 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.721268 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.721843 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.721987 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.726595 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.726601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.726605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.726642 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.729533 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.730419 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.730425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.730429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.730636 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.731110 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.731115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.731118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.731406 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.731574 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.732330 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.732572 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":1092,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:40:32.734133 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.712898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40001 (local address 127.3.85.130:33637)
0619 18:40:32.713768 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.713817 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.716132 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.716995 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.717005 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.717652 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:32.718504 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.718530 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.719740 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.719750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.721086 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.721101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.722124 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.722133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.722415 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.723317 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:32.723369 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:32.726277 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:32.729732 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.729743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.729770 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.730257 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.730757 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.730762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.730766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.730985 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.731555 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:32.731564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:32.731949 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.732235 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.732844 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":362,"threads_started":1}
I20260619 18:40:32.735863 3790 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
I20260619 18:40:32.736892 3790 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:32.737834 3790 heartbeater.cc:511] Master 127.3.85.190:40001 requested a full tablet report, sending...
I20260619 18:40:32.739683 3466 ts_manager.cc:194] Registered new tserver with Master: 5747a50c7cbf430c9ec5dbf65921e0f1 (127.3.85.130:40929)
I20260619 18:40:32.740551 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:33637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:40:32.761436 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:33.121938 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:33.122262 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:33.122437 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:33.130661 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:33.130781 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:33.131008 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:40:33.141651 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40001
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:33.143214 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:33.144670 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:33.159334 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:33.159279 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:33.160598 3796 server_base.cc:1034] running on GCE node
W20260619 18:40:33.162046 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:33.162864 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:33.163918 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:33.165594 3796 hybrid_clock.cc:648] HybridClock initialized: now 1781894433165117 us; error 52 us; skew 500 ppm
I20260619 18:40:33.165967 3796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:33.169409 3796 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:40:33.171147 3796 webserver.cc:460] Webserver started at http://127.3.85.131:42469/ using document root <none> and password file <none>
I20260619 18:40:33.171887 3796 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:33.172008 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:33.172322 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:33.175632 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "2407aab9face4c64a833331de0c9e9b8"
format_stamp: "Formatted at 2026-06-19 18:40:33 on dist-test-slave-09fp"
server_key: "7e0e2c567306a98e985b4c34616a6d19"
server_key_iv: "90fc12f184b0bcb5f7d741731446338e"
server_key_version: "encryptionkey@0"
I20260619 18:40:33.176451 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "2407aab9face4c64a833331de0c9e9b8"
format_stamp: "Formatted at 2026-06-19 18:40:33 on dist-test-slave-09fp"
server_key: "7e0e2c567306a98e985b4c34616a6d19"
server_key_iv: "90fc12f184b0bcb5f7d741731446338e"
server_key_version: "encryptionkey@0"
I20260619 18:40:33.182384 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260619 18:40:33.187408 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:33.189229 3796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:40:33.189467 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "2407aab9face4c64a833331de0c9e9b8"
format_stamp: "Formatted at 2026-06-19 18:40:33 on dist-test-slave-09fp"
server_key: "7e0e2c567306a98e985b4c34616a6d19"
server_key_iv: "90fc12f184b0bcb5f7d741731446338e"
server_key_version: "encryptionkey@0"
I20260619 18:40:33.189718 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:33.196583 3656 heartbeater.cc:503] Master 127.3.85.190:40001 was elected leader, sending a full tablet report...
I20260619 18:40:33.213001 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:33.217928 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:33.218272 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:33.219458 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:33.220902 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:33.221038 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:33.221201 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:33.221280 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:33.259680 3796 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:36711
I20260619 18:40:33.259794 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:36711 every 8 connection(s)
I20260619 18:40:33.261165 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:33.270884 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 3796
I20260619 18:40:33.271081 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:33.271534 3414 external_mini_cluster.cc:1428] Setting key 5424067c592c83a4b271661e4b404733
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:33.284396 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.264029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:46541 (local address 127.3.85.190:40001)
0619 18:40:33.264515 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.264522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.265772 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 18:40:33.267823 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.267827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.267903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.267999 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.269864 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.270730 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.272609 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.273195 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.273352 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.277786 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.277792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.277795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.277833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.280515 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.281390 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.281395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.281398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.281955 (+ 557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.282487 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.282492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.282496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.282814 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.282975 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.283734 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.283991 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260619 18:40:33.285521 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.264694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40001 (local address 127.3.85.131:46541)
0619 18:40:33.265577 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:33.265623 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:33.267489 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:33.268315 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:33.268326 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:33.268941 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:33.269618 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:33.269648 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.270918 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.270922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:33.272438 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:33.272454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.273453 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.273462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:33.273722 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.275102 (+ 1380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:33.275128 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:33.277490 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:33.280707 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:33.280718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:33.280740 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:33.281225 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:33.282099 (+ 874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:33.282104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:33.282108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:33.282338 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:33.282952 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:33.282961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:33.283359 (+ 398us) client_negotiation.cc:769] Sending connection context
0619 18:40:33.283648 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 18:40:33.284285 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":332,"threads_started":1}
I20260619 18:40:33.287359 3924 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40001
I20260619 18:40:33.287741 3924 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:33.288658 3924 heartbeater.cc:511] Master 127.3.85.190:40001 requested a full tablet report, sending...
I20260619 18:40:33.290591 3465 ts_manager.cc:194] Registered new tserver with Master: 2407aab9face4c64a833331de0c9e9b8 (127.3.85.131:36711)
I20260619 18:40:33.291487 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:46541
I20260619 18:40:33.300921 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:33.349691 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.332217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.3.85.129:45889)
0619 18:40:33.332902 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.332910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.332940 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:40:33.333123 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.333130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.333369 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.333508 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.335063 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.336138 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.337586 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.338353 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.338562 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.343348 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.343365 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.343382 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.343435 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.346582 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.347406 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.347413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.347416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.347556 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.347987 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.347993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.347996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.348372 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.348695 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.349026 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.349302 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":402,"threads_started":1}
W20260619 18:40:33.351184 3571 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45476
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:33.398877 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.383570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.3.85.129:45889)
0619 18:40:33.383835 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.383840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.384061 (+ 221us) server_negotiation.cc:407] Connection header received
0619 18:40:33.384223 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.384227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.384324 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.384417 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.385850 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.386745 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.388160 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.388644 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.388751 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.393395 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.393401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.393404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.393447 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.396292 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.397005 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.397010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.397012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.397148 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.397493 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.397496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.397498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.397753 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.397925 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.398270 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.398438 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 18:40:33.400686 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 3528
I20260619 18:40:33.414618 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.401730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.3.85.129:45889)
0619 18:40:33.401913 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.401917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.402389 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:40:33.402584 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.402588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.402652 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.402743 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.404049 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.404837 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.406174 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.406748 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.406879 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.409218 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.409223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.409227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.409265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.411963 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.412729 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.412734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.412736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.412891 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.413267 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.413269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.413271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.413522 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.413717 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.414056 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.414243 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:33.415347 3651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:33.504571 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 3528
I20260619 18:40:33.521776 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 3662
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:40:33.539400 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.522807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.3.85.130:40929)
0619 18:40:33.523329 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.523336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.523363 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:40:33.523432 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.523439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.523672 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.523811 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.524996 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.526001 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.527562 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.528256 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.528368 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.532615 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.532630 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.532666 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.532715 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.535914 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.536861 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.536878 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.536880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.537061 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.537525 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.537529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.537531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.537960 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.538286 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.538699 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.538921 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":263,"threads_started":1}
I20260619 18:40:33.541098 3785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:33.625834 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 3662
I20260619 18:40:33.648308 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 3796
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:40:33.665342 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.649354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.3.85.131:36711)
0619 18:40:33.649936 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:33.649944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:33.649971 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:40:33.650030 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:33.650036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:33.650253 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:33.650429 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:33.651790 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.652848 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.654191 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:33.654999 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:33.655155 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:33.659539 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:33.659554 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:33.659570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:33.659613 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:33.662316 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.663133 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.663141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.663143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.663271 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:33.663672 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:33.663688 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:33.663690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:33.664041 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:33.664359 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:33.664704 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:40:33.664991 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":292,"threads_started":1}
I20260619 18:40:33.666688 3919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:33.750526 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 3796
I20260619 18:40:33.767899 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 3434
I20260619 18:40:33.769305 3496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:33.871562 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 3434
2026-06-19T18:40:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4185 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3946](info): set up 2 sockets
Jun 19 18:40:33 dist-test-slave-09fp krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260619 18:40:34.959151 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:35 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:35.146843 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46585
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:35.482555 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:35.482893 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:35.482985 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:35.490612 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:35.490741 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:35.490818 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:35.490911 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:35.490999 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:35.501386 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:35.502895 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:35.504410 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:35.517849 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:35.517872 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:35.517910 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:35.518958 3959 server_base.cc:1034] running on GCE node
I20260619 18:40:35.519882 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:35.521489 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:35.522743 3959 hybrid_clock.cc:648] HybridClock initialized: now 1781894435522684 us; error 58 us; skew 500 ppm
I20260619 18:40:35.523130 3959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:35 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894435, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:35.530336 3959 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:35.532176 3959 webserver.cc:460] Webserver started at http://127.3.85.190:39671/ using document root <none> and password file <none>
I20260619 18:40:35.532924 3959 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:35.533035 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:35.533303 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:35.536525 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "38c643c5d8dc437e87789d3fe2d53897"
format_stamp: "Formatted at 2026-06-19 18:40:35 on dist-test-slave-09fp"
server_key: "6ad6ed34371d52105f036e7f25bd7979"
server_key_iv: "b79503664a964a22b4ad9f2bc37430d7"
server_key_version: "encryptionkey@0"
I20260619 18:40:35.537290 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "38c643c5d8dc437e87789d3fe2d53897"
format_stamp: "Formatted at 2026-06-19 18:40:35 on dist-test-slave-09fp"
server_key: "6ad6ed34371d52105f036e7f25bd7979"
server_key_iv: "b79503664a964a22b4ad9f2bc37430d7"
server_key_version: "encryptionkey@0"
I20260619 18:40:35.543167 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 18:40:35.547266 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:35.549099 3959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 18:40:35.549351 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "38c643c5d8dc437e87789d3fe2d53897"
format_stamp: "Formatted at 2026-06-19 18:40:35 on dist-test-slave-09fp"
server_key: "6ad6ed34371d52105f036e7f25bd7979"
server_key_iv: "b79503664a964a22b4ad9f2bc37430d7"
server_key_version: "encryptionkey@0"
I20260619 18:40:35.549666 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:35.566071 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:35.570974 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:35.571290 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:35.591216 3959 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46585
I20260619 18:40:35.591235 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46585 every 8 connection(s)
I20260619 18:40:35.592715 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:35.595553 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 3959
I20260619 18:40:35.595783 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:35.596614 3414 external_mini_cluster.cc:1428] Setting key 40fcc71e1d37783a752944550f975353
I20260619 18:40:35.597533 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:40:35 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:35.613574 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897: Bootstrap starting.
I20260619 18:40:35.618316 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:35.619697 4026 log.cc:824] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:35.623137 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897: No bootstrap required, opened a new log
I20260619 18:40:35.623230 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:35.598907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.3.85.190:46585)
0619 18:40:35.599742 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:35.599756 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:35.599801 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:40:35.601404 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:35.601429 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:35.601918 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:35.602349 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:35.604620 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:35.606156 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:35.608030 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:35.608825 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:35.608983 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:35.614040 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:35.614053 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:35.614077 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:35.614136 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:35.618256 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:35.619201 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:35.619210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:35.619222 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:35.619493 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:35.619954 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:35.619958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:35.619960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:35.620628 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:35.620998 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:35.621551 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 18:40:35.622019 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":387,"threads_started":1}
I20260619 18:40:35.631460 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } }
I20260619 18:40:35.631935 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:35.632051 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c643c5d8dc437e87789d3fe2d53897, State: Initialized, Role: FOLLOWER
I20260619 18:40:35.632769 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } }
I20260619 18:40:35.633044 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:35.633199 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:35.633391 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:35.635717 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } }
I20260619 18:40:35.636286 4026 leader_election.cc:302] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c643c5d8dc437e87789d3fe2d53897; no voters:
I20260619 18:40:35.636994 4026 leader_election.cc:288] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:35.637113 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:35.638089 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [term 1 LEADER]: Becoming Leader. State: Replica: 38c643c5d8dc437e87789d3fe2d53897, State: Running, Role: LEADER
I20260619 18:40:35.638710 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } }
I20260619 18:40:35.639510 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:35.641995 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c643c5d8dc437e87789d3fe2d53897. Latest consensus state: current_term: 1 leader_uuid: "38c643c5d8dc437e87789d3fe2d53897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } } }
I20260619 18:40:35.641970 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38c643c5d8dc437e87789d3fe2d53897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c643c5d8dc437e87789d3fe2d53897" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46585 } } }
I20260619 18:40:35.642488 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:35.642532 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:35.649179 4037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:35.656169 4037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:35.666658 4037 catalog_manager.cc:1347] Generated new cluster ID: 1947eb6a00a440aca558cc1eeeea4536
I20260619 18:40:35.666831 4037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:35.691097 4037 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:35.692543 4037 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:35.701179 4037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38c643c5d8dc437e87789d3fe2d53897: Generated new TSK 0
I20260619 18:40:35.702127 4037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:40:35.754431 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:36.102046 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:36.102345 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:36.102512 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:36.110088 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:36.110222 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:36.110420 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:40:36.120648 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:36.122025 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:36.123476 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:36.137290 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:36.137212 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:36.137259 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:36.138126 4052 server_base.cc:1034] running on GCE node
I20260619 18:40:36.138933 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:36.140110 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:36.141361 4052 hybrid_clock.cc:648] HybridClock initialized: now 1781894436141325 us; error 59 us; skew 500 ppm
I20260619 18:40:36.141748 4052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:36 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:36.145177 4052 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:40:36.146782 4052 webserver.cc:460] Webserver started at http://127.3.85.129:34429/ using document root <none> and password file <none>
I20260619 18:40:36.147526 4052 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:36.147630 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:36.147902 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:36.151129 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "89b64e8f2d834d158d7d4114982395be"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "243c8cc5b7ef949df0263923e95913c6"
server_key_iv: "9b01e11f67851a40672391e5abada3af"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.151971 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "89b64e8f2d834d158d7d4114982395be"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "243c8cc5b7ef949df0263923e95913c6"
server_key_iv: "9b01e11f67851a40672391e5abada3af"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.157783 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:40:36.161702 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.163334 4052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:36.163538 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "89b64e8f2d834d158d7d4114982395be"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "243c8cc5b7ef949df0263923e95913c6"
server_key_iv: "9b01e11f67851a40672391e5abada3af"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.163753 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:36.187386 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:36.192644 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:36.192972 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:36.194084 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:36.195643 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:36.195724 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.195856 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:36.195922 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.230757 4052 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:43199
I20260619 18:40:36.230842 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:43199 every 8 connection(s)
I20260619 18:40:36.232244 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:36.233582 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4052
I20260619 18:40:36.233804 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:36.234606 3414 external_mini_cluster.cc:1428] Setting key 0e16a6ef9dc5beb7da0c1309c37339ec
Jun 19 18:40:36 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:40:36.258697 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:36.235419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:60181 (local address 127.3.85.190:46585)
0619 18:40:36.235911 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:36.235919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:36.237325 (+ 1406us) server_negotiation.cc:407] Connection header received
0619 18:40:36.239717 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:36.239726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:36.239821 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:36.239944 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:36.242514 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.243241 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.246428 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.246919 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.247037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:36.252260 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:36.252266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:36.252269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:36.252308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:36.255253 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:36.256106 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:36.256114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:36.256117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:36.256253 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:36.256797 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:36.256803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:36.256808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:36.257091 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:36.257248 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:36.258055 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 18:40:36.258287 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260619 18:40:36.259783 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:36.236124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46585 (local address 127.3.85.129:60181)
0619 18:40:36.237122 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:36.237184 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:36.239297 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:36.240237 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:36.240246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:36.241136 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:36.242199 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:36.242233 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.243442 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.243452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:36.246204 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:36.246228 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.247232 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.247244 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:36.247680 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:36.248616 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:36.248661 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:36.251975 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:36.255433 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:36.255444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:36.255481 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:36.255960 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:36.256389 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:36.256394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:36.256398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:36.256675 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:36.257238 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:36.257246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:36.257656 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 18:40:36.257948 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 18:40:36.258569 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":428,"threads_started":1}
I20260619 18:40:36.261610 4180 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46585
I20260619 18:40:36.261960 4180 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:36.262780 4180 heartbeater.cc:511] Master 127.3.85.190:46585 requested a full tablet report, sending...
I20260619 18:40:36.265064 3991 ts_manager.cc:194] Registered new tserver with Master: 89b64e8f2d834d158d7d4114982395be (127.3.85.129:43199)
I20260619 18:40:36.266662 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:60181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:40:36.281366 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:36.617522 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:36.617830 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:36.617928 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:36.625449 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:36.625624 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:36.625751 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:40:36.637629 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:36.639037 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:36.640599 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:36.655853 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:36.655965 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:36.655853 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:36.657064 4186 server_base.cc:1034] running on GCE node
I20260619 18:40:36.657773 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:36.659350 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:36.660604 4186 hybrid_clock.cc:648] HybridClock initialized: now 1781894436660534 us; error 72 us; skew 500 ppm
I20260619 18:40:36.660948 4186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:36 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:36.664228 4186 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:40:36.665844 4186 webserver.cc:460] Webserver started at http://127.3.85.130:39917/ using document root <none> and password file <none>
I20260619 18:40:36.666718 4186 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:36.666872 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:36.667229 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:36.670634 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "cb631d025a594b5aa6b74fad7d4b0f22"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "16d8aace5c3f999374bc3f28cbd1cca4"
server_key_iv: "19652ee0814fd404a2da1d5ebe4755db"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.671512 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "cb631d025a594b5aa6b74fad7d4b0f22"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "16d8aace5c3f999374bc3f28cbd1cca4"
server_key_iv: "19652ee0814fd404a2da1d5ebe4755db"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.677208 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:40:36.681269 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.682960 4186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:40:36.683235 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "cb631d025a594b5aa6b74fad7d4b0f22"
format_stamp: "Formatted at 2026-06-19 18:40:36 on dist-test-slave-09fp"
server_key: "16d8aace5c3f999374bc3f28cbd1cca4"
server_key_iv: "19652ee0814fd404a2da1d5ebe4755db"
server_key_version: "encryptionkey@0"
I20260619 18:40:36.683480 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:36.707309 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:36.714238 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:36.714555 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:36.715557 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:36.716892 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:36.716972 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.717067 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:36.717113 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:36.751466 4186 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:42035
I20260619 18:40:36.751554 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:42035 every 8 connection(s)
I20260619 18:40:36.752853 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:36.760046 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4186
I20260619 18:40:36.760293 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:36.760780 3414 external_mini_cluster.cc:1428] Setting key 3cf280e47615b3b95e961502e1fbe68e
Jun 19 18:40:36 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894436, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:36.776330 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:36.755771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45111 (local address 127.3.85.190:46585)
0619 18:40:36.756005 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:36.756011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:36.757248 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 18:40:36.759201 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:36.759206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:36.759279 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:36.759373 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:36.761205 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.761993 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.763980 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.764815 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.764973 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:36.769405 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:36.769411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:36.769414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:36.769450 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:36.772147 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:36.773326 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:36.773330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:36.773332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:36.773469 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:36.774171 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:36.774177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:36.774181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:36.774605 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:36.774736 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:36.775859 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0619 18:40:36.776063 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 18:40:36.778076 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:36.756299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46585 (local address 127.3.85.130:45111)
0619 18:40:36.757060 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:36.757110 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:36.758908 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:36.759606 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:36.759615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:36.760179 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:36.761001 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:36.761028 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.762183 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.762189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:36.763815 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:36.763828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:36.765068 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:36.765073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:36.765297 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:36.766092 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:36.766117 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:36.769102 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:36.772353 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:36.772367 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:36.772401 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:36.773165 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:36.773602 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:36.773610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:36.773617 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:36.773920 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:36.774765 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:36.774775 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:36.775359 (+ 584us) client_negotiation.cc:769] Sending connection context
0619 18:40:36.775752 (+ 393us) client_negotiation.cc:240] Negotiation successful
0619 18:40:36.776474 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:40:36.779819 4314 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46585
I20260619 18:40:36.780164 4314 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:36.780895 4314 heartbeater.cc:511] Master 127.3.85.190:46585 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
I20260619 18:40:36.782583 3991 ts_manager.cc:194] Registered new tserver with Master: cb631d025a594b5aa6b74fad7d4b0f22 (127.3.85.130:42035)
I20260619 18:40:36.783414 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:45111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:40:36.803974 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:37.152940 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:37.153239 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:37.153347 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:37.161085 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:37.161226 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:37.161352 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:40:37.172358 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:37.173663 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:37.175143 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:37.190341 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:37.190647 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:37.190943 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:37.191730 4319 server_base.cc:1034] running on GCE node
I20260619 18:40:37.192503 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:37.193689 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:37.194928 4319 hybrid_clock.cc:648] HybridClock initialized: now 1781894437194876 us; error 54 us; skew 500 ppm
I20260619 18:40:37.195297 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:37.198880 4319 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:40:37.200589 4319 webserver.cc:460] Webserver started at http://127.3.85.131:35249/ using document root <none> and password file <none>
I20260619 18:40:37.201397 4319 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:37.201565 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:37.201908 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:37.205314 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "567c6b9eb6e046f5a4a74026dd94b37e"
format_stamp: "Formatted at 2026-06-19 18:40:37 on dist-test-slave-09fp"
server_key: "ead67062b2238ecf1ef0dc6cd77a0059"
server_key_iv: "308fda65a9fb55a84e5c7030183196ae"
server_key_version: "encryptionkey@0"
I20260619 18:40:37.206239 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "567c6b9eb6e046f5a4a74026dd94b37e"
format_stamp: "Formatted at 2026-06-19 18:40:37 on dist-test-slave-09fp"
server_key: "ead67062b2238ecf1ef0dc6cd77a0059"
server_key_iv: "308fda65a9fb55a84e5c7030183196ae"
server_key_version: "encryptionkey@0"
I20260619 18:40:37.212230 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 18:40:37.216353 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:37.217994 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:37.218212 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "567c6b9eb6e046f5a4a74026dd94b37e"
format_stamp: "Formatted at 2026-06-19 18:40:37 on dist-test-slave-09fp"
server_key: "ead67062b2238ecf1ef0dc6cd77a0059"
server_key_iv: "308fda65a9fb55a84e5c7030183196ae"
server_key_version: "encryptionkey@0"
I20260619 18:40:37.218448 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:37.241778 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:37.246340 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:37.246687 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:37.247762 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:37.249670 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:37.249780 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:37.249935 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:37.250017 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:37.270418 4180 heartbeater.cc:503] Master 127.3.85.190:46585 was elected leader, sending a full tablet report...
I20260619 18:40:37.285391 4319 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46053
I20260619 18:40:37.285475 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46053 every 8 connection(s)
I20260619 18:40:37.286847 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:37.295444 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4319
I20260619 18:40:37.295660 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:37.296209 3414 external_mini_cluster.cc:1428] Setting key c0fc5a489809a4e534daf646fd502a73
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:37.308722 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.289617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:38559 (local address 127.3.85.190:46585)
0619 18:40:37.290101 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.290107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.291158 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 18:40:37.293015 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.293020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.293092 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.293198 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.294889 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.295703 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.297868 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.298319 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.298463 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.302792 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.302798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.302801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.302840 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.305551 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.306359 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.306363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.306366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.306591 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.307027 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.307030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.307032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.307309 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.307432 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.308154 (+ 722us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.308342 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
I20260619 18:40:37.309700 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.290180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46585 (local address 127.3.85.131:38559)
0619 18:40:37.290975 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.291028 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.292689 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.293438 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.293447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.293998 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:37.294692 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.294719 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.295925 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.295932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.297687 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.297727 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.298613 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.298621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.298946 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.299786 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:37.299811 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:37.302074 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:37.305730 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:37.305738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:37.305754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:37.306230 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:37.306694 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:37.306697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:37.306700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:37.306890 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:37.307419 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:37.307426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:37.307788 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.308036 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.308465 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260619 18:40:37.311393 4447 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46585
I20260619 18:40:37.311784 4447 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:37.312726 4447 heartbeater.cc:511] Master 127.3.85.190:46585 requested a full tablet report, sending...
I20260619 18:40:37.314381 3990 ts_manager.cc:194] Registered new tserver with Master: 567c6b9eb6e046f5a4a74026dd94b37e (127.3.85.131:46053)
I20260619 18:40:37.315472 3990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:38559
I20260619 18:40:37.325321 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:37.346086 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.332173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.3.85.190:46585)
0619 18:40:37.332359 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.332363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.332979 (+ 616us) server_negotiation.cc:407] Connection header received
0619 18:40:37.333137 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.333141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.333212 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.333377 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.335171 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.336491 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.337932 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.338364 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.338579 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.340970 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.340976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.340979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.341019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.343702 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.344401 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.344405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.344407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.344529 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.344857 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.344860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.344862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.345196 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.345373 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.345660 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.345835 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:40:37.353376 3990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:37.361110 3990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:37.399709 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.375066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:42035 (local address 127.0.0.1:36426)
0619 18:40:37.385788 (+ 10722us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.385860 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.386191 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.391824 (+ 5633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.391833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.391873 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:37.392373 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.392387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.394315 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.394322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.396814 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.396830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.398637 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.398643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.398850 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.398925 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.399081 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.399300 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10313,"thread_start_us":16094,"threads_started":1}
I20260619 18:40:37.400832 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.375134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.3.85.130:42035)
0619 18:40:37.390245 (+ 15111us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.390258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.390305 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:40:37.390476 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.390489 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.391241 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.391492 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:37.392550 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.394120 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.397012 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.398362 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.398562 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.398799 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.400293 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.400483 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14871,"thread_start_us":17364,"threads_started":1}
I20260619 18:40:37.401419 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.375125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:43199 (local address 127.0.0.1:59140)
0619 18:40:37.390919 (+ 15794us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.390943 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.391077 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.392079 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.392091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.392135 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:37.392758 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.392778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.395501 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.395507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.397425 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.397439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.400369 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.400374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.400575 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.400600 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.400686 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.400888 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15639,"thread_start_us":17763,"threads_started":1}
I20260619 18:40:37.401865 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.375795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.3.85.129:43199)
0619 18:40:37.391399 (+ 15604us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.391410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.391441 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:40:37.391553 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.391564 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.391823 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.392047 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:37.393033 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.395286 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.397750 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.399775 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.401116 (+ 1341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.401294 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.401359 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.401530 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15413,"thread_start_us":16401,"threads_started":1}
I20260619 18:40:37.402549 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.375899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46053 (local address 127.0.0.1:43356)
0619 18:40:37.393215 (+ 17316us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.393235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.393365 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.393980 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.393985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.394013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:37.394474 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.394486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.396532 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.396538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.398400 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.398412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.400802 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.400808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.401750 (+ 942us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.401789 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.401887 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.402204 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17171,"thread_start_us":9253,"threads_started":1}
I20260619 18:40:37.402575 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.376694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.3.85.131:46053)
0619 18:40:37.387140 (+ 10446us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.387155 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.393376 (+ 6221us) server_negotiation.cc:407] Connection header received
0619 18:40:37.393499 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.393508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.393762 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.393972 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:37.394710 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.396344 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.399081 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.400587 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.400726 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.400986 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.401992 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.402173 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10151,"thread_start_us":9956,"threads_started":1}
I20260619 18:40:37.405577 4249 tablet_service.cc:1467] Processing CreateTablet for tablet d3794cb15f664cccbf3e815b3d6343f5 (DEFAULT_TABLE table=test-table [id=9cf0cfac5e104ccb88a5ee5f519d43b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:37.405910 4115 tablet_service.cc:1467] Processing CreateTablet for tablet d3794cb15f664cccbf3e815b3d6343f5 (DEFAULT_TABLE table=test-table [id=9cf0cfac5e104ccb88a5ee5f519d43b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:37.407307 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3794cb15f664cccbf3e815b3d6343f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:37.407334 4249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3794cb15f664cccbf3e815b3d6343f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:37.406761 4382 tablet_service.cc:1467] Processing CreateTablet for tablet d3794cb15f664cccbf3e815b3d6343f5 (DEFAULT_TABLE table=test-table [id=9cf0cfac5e104ccb88a5ee5f519d43b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:37.408321 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3794cb15f664cccbf3e815b3d6343f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:37.421175 4466 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Bootstrap starting.
I20260619 18:40:37.423681 4468 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Bootstrap starting.
I20260619 18:40:37.425885 4467 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Bootstrap starting.
I20260619 18:40:37.425956 4466 tablet_bootstrap.cc:654] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:37.427543 4468 tablet_bootstrap.cc:654] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:37.427733 4466 log.cc:824] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:37.429600 4467 tablet_bootstrap.cc:654] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:37.429627 4468 log.cc:824] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:37.431216 4467 log.cc:824] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:37.433881 4467 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: No bootstrap required, opened a new log
I20260619 18:40:37.433930 4468 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: No bootstrap required, opened a new log
I20260619 18:40:37.434250 4467 ts_tablet_manager.cc:1397] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 18:40:37.434266 4468 ts_tablet_manager.cc:1397] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260619 18:40:37.437170 4466 tablet_bootstrap.cc:492] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: No bootstrap required, opened a new log
I20260619 18:40:37.437927 4466 ts_tablet_manager.cc:1397] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260619 18:40:37.442615 4467 raft_consensus.cc:348] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.443068 4467 raft_consensus.cc:374] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:37.443166 4467 raft_consensus.cc:729] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb631d025a594b5aa6b74fad7d4b0f22, State: Initialized, Role: FOLLOWER
I20260619 18:40:37.443871 4467 consensus_queue.cc:260] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.445099 4314 heartbeater.cc:503] Master 127.3.85.190:46585 was elected leader, sending a full tablet report...
I20260619 18:40:37.445706 4467 ts_tablet_manager.cc:1428] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 18:40:37.445765 4468 raft_consensus.cc:348] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.446254 4468 raft_consensus.cc:374] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:37.446844 4468 raft_consensus.cc:729] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567c6b9eb6e046f5a4a74026dd94b37e, State: Initialized, Role: FOLLOWER
I20260619 18:40:37.447625 4468 consensus_queue.cc:260] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.448905 4447 heartbeater.cc:503] Master 127.3.85.190:46585 was elected leader, sending a full tablet report...
I20260619 18:40:37.450117 4468 ts_tablet_manager.cc:1428] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Time spent starting tablet: real 0.016s user 0.009s sys 0.009s
I20260619 18:40:37.450953 4466 raft_consensus.cc:348] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.451476 4466 raft_consensus.cc:374] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:37.451607 4466 raft_consensus.cc:729] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b64e8f2d834d158d7d4114982395be, State: Initialized, Role: FOLLOWER
I20260619 18:40:37.452334 4466 consensus_queue.cc:260] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.455219 4466 ts_tablet_manager.cc:1428] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Time spent starting tablet: real 0.017s user 0.017s sys 0.001s
W20260619 18:40:37.484812 4181 tablet.cc:2369] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:37.505043 4315 tablet.cc:2369] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:37.538775 4448 tablet.cc:2369] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:37.702504 4473 raft_consensus.cc:482] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:37.702853 4473 raft_consensus.cc:504] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.704669 4473 leader_election.cc:288] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb631d025a594b5aa6b74fad7d4b0f22 (127.3.85.130:42035), 89b64e8f2d834d158d7d4114982395be (127.3.85.129:43199)
I20260619 18:40:37.711638 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.704914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:42035 (local address 127.3.85.131:54893)
0619 18:40:37.705138 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.705164 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.705299 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.705758 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.705763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.705800 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:37.706005 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.706014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.707442 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.707446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.709492 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.709509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.710881 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.710886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.711113 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.711150 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.711244 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.711349 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:40:37.711725 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.705049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:54893 (local address 127.3.85.130:42035)
0619 18:40:37.705276 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.705281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.705303 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:40:37.705514 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.705519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.705600 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.705694 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:37.706221 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.707266 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.709687 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.710721 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.710877 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.710956 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.711297 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.711452 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:40:37.712384 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.705425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:55249 (local address 127.3.85.129:43199)
0619 18:40:37.705656 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.705664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.706231 (+ 567us) server_negotiation.cc:407] Connection header received
0619 18:40:37.706442 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.706479 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.706562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.706655 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:37.707230 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.708053 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.710229 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.711394 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.711561 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.711637 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.711991 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.712152 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:40:37.712927 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.705218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:43199 (local address 127.3.85.131:55249)
0619 18:40:37.706100 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:37.706126 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:37.706313 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:37.706731 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:37.706736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:37.706767 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:37.707075 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.707084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.708202 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.708209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.710065 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:37.710075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.711598 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.711604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:37.711802 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.711837 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:40:37.712566 (+ 729us) client_negotiation.cc:240] Negotiation successful
0619 18:40:37.712697 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":445,"threads_started":1}
I20260619 18:40:37.712932 4269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3794cb15f664cccbf3e815b3d6343f5" candidate_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" is_pre_election: true
I20260619 18:40:37.713567 4269 raft_consensus.cc:2454] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567c6b9eb6e046f5a4a74026dd94b37e in term 0.
I20260619 18:40:37.713936 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3794cb15f664cccbf3e815b3d6343f5" candidate_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b64e8f2d834d158d7d4114982395be" is_pre_election: true
I20260619 18:40:37.714534 4135 raft_consensus.cc:2454] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567c6b9eb6e046f5a4a74026dd94b37e in term 0.
I20260619 18:40:37.714815 4338 leader_election.cc:302] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567c6b9eb6e046f5a4a74026dd94b37e, cb631d025a594b5aa6b74fad7d4b0f22; no voters:
I20260619 18:40:37.715260 4473 raft_consensus.cc:2781] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:37.715401 4473 raft_consensus.cc:482] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:37.715538 4473 raft_consensus.cc:3037] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:37.717720 4473 raft_consensus.cc:504] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.718423 4473 leader_election.cc:288] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [CANDIDATE]: Term 1 election: Requested vote from peers cb631d025a594b5aa6b74fad7d4b0f22 (127.3.85.130:42035), 89b64e8f2d834d158d7d4114982395be (127.3.85.129:43199)
I20260619 18:40:37.718930 4269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3794cb15f664cccbf3e815b3d6343f5" candidate_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb631d025a594b5aa6b74fad7d4b0f22"
I20260619 18:40:37.719002 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3794cb15f664cccbf3e815b3d6343f5" candidate_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89b64e8f2d834d158d7d4114982395be"
I20260619 18:40:37.719188 4269 raft_consensus.cc:3037] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:37.719198 4135 raft_consensus.cc:3037] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:37.722599 4269 raft_consensus.cc:2454] T d3794cb15f664cccbf3e815b3d6343f5 P cb631d025a594b5aa6b74fad7d4b0f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567c6b9eb6e046f5a4a74026dd94b37e in term 1.
I20260619 18:40:37.722599 4135 raft_consensus.cc:2454] T d3794cb15f664cccbf3e815b3d6343f5 P 89b64e8f2d834d158d7d4114982395be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567c6b9eb6e046f5a4a74026dd94b37e in term 1.
I20260619 18:40:37.723178 4337 leader_election.cc:302] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567c6b9eb6e046f5a4a74026dd94b37e, 89b64e8f2d834d158d7d4114982395be; no voters:
I20260619 18:40:37.723557 4473 raft_consensus.cc:2781] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:37.723984 4473 raft_consensus.cc:686] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [term 1 LEADER]: Becoming Leader. State: Replica: 567c6b9eb6e046f5a4a74026dd94b37e, State: Running, Role: LEADER
I20260619 18:40:37.724589 4473 consensus_queue.cc:237] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } }
I20260619 18:40:37.729377 3990 catalog_manager.cc:5495] T d3794cb15f664cccbf3e815b3d6343f5 P 567c6b9eb6e046f5a4a74026dd94b37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 567c6b9eb6e046f5a4a74026dd94b37e (127.3.85.131). New cstate: current_term: 1 leader_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb631d025a594b5aa6b74fad7d4b0f22" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89b64e8f2d834d158d7d4114982395be" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567c6b9eb6e046f5a4a74026dd94b37e" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46053 } health_report { overall_health: HEALTHY } } }
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:37.776119 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.759327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.3.85.129:43199)
0619 18:40:37.759724 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.759741 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.760159 (+ 418us) server_negotiation.cc:407] Connection header received
0619 18:40:37.760382 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.760393 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.760488 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.760601 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.762185 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.763010 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.764348 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.764788 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.764947 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.769251 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.769268 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.769285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.769330 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.773013 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.773821 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.773828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.773831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.773993 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.774399 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.774405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.774409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.774860 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.775169 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.775502 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.775727 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:37.827718 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.811643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.3.85.129:43199)
0619 18:40:37.811884 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.811893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.812254 (+ 361us) server_negotiation.cc:407] Connection header received
0619 18:40:37.812529 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.812533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.812600 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.812708 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.814230 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.815088 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.816397 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.816810 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.816955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.821104 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.821110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.821113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.821153 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.824060 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.825377 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.825382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.825385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.825564 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.826143 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.826148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.826151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.826562 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.826860 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.827258 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.827466 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:40:37 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:37.875422 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.860454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.3.85.129:43199)
0619 18:40:37.860752 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.860759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.860923 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:40:37.861080 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.861084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.861146 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.861230 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.862617 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.863335 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.864594 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.865034 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.865137 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.869597 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.869604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.869607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.869650 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.872418 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.873248 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.873252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.873255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.873385 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.873756 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.873759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.873761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.874057 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.874258 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.874968 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.875160 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260619 18:40:37.877272 4095 tablet_service.cc:2908] Rejecting scan request for tablet d3794cb15f664cccbf3e815b3d6343f5: Uninitialized: clean time has not yet been initialized
I20260619 18:40:37.880538 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4052
I20260619 18:40:37.894591 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:37.881594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.3.85.129:43199)
0619 18:40:37.881775 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:37.881779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:37.882148 (+ 369us) server_negotiation.cc:407] Connection header received
0619 18:40:37.882325 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:37.882329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:37.882404 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:37.882562 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:37.883773 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.884512 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.885747 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:37.886149 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:37.886268 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:37.888915 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:37.888920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:37.888924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:37.888960 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:37.891936 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.892745 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.892749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.892752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.892906 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:37.893300 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:37.893304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:37.893307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:37.893624 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:37.893851 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:37.894085 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 18:40:37.894264 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:40:37.895249 4175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:37.986296 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4052
I20260619 18:40:38.003770 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4186
Jun 19 18:40:38 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:40:38.019940 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:38.004630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36436 (local address 127.3.85.130:42035)
0619 18:40:38.004831 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:38.004837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:38.004857 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:40:38.005029 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:38.005035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:38.005124 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:38.005237 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:38.006573 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:38.007312 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:38.009118 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:38.009521 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:38.009624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:38.014194 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:38.014211 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:38.014227 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:38.014275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:38.017115 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:38.017914 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:38.017922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:38.017924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:38.018057 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:38.018438 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:38.018441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:38.018443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:38.018883 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:38.019111 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:38.019399 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:40:38.019563 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:40:38.020644 4309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:38.109411 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4186
I20260619 18:40:38.126677 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4319
Jun 19 18:40:38 dist-test-slave-09fp krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:40:38.142253 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:38.141711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:42035 (local address 127.3.85.131:41777)
0619 18:40:38.141911 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:38.142035 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.130:42035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:40:38.142596 4338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.130:42035: connect: Connection refused (error 111)
I20260619 18:40:38.142975 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:38.127496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.3.85.131:46053)
0619 18:40:38.127994 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:38.128000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:38.128019 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:40:38.128083 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:38.128086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:38.128151 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:38.128249 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:38.129461 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:38.130175 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:38.131577 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:38.132009 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:38.132126 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:38.136555 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:38.136571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:38.136588 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:38.136632 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:38.139719 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:38.140521 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:38.140530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:38.140532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:38.140743 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:38.141179 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:38.141182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:38.141184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:38.141620 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:38.141924 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:38.142228 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 18:40:38.142507 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
I20260619 18:40:38.143667 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:38.233722 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4319
I20260619 18:40:38.250675 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 3959
I20260619 18:40:38.251917 4021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:38.360067 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 3959
2026-06-19T18:40:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4488 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:38 dist-test-slave-09fp krb5kdc[4497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:38 dist-test-slave-09fp krb5kdc[4497](info): set up 2 sockets
Jun 19 18:40:38 dist-test-slave-09fp krb5kdc[4497](info): commencing operation
krb5kdc: starting...
W20260619 18:40:39.454903 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:39 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:39.655349 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43591
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:39.993458 4510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:39.993775 4510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:39.993947 4510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:40.001441 4510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:40.001570 4510 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:40:40.001652 4510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:40.001750 4510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:40.001840 4510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:40.001925 4510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:40.012750 4510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:43591
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:40.014201 4510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:40.015720 4510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:40.029492 4518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:40.029458 4515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:40.029425 4516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:40.030803 4510 server_base.cc:1034] running on GCE node
I20260619 18:40:40.031669 4510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:40.033299 4510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:40.034634 4510 hybrid_clock.cc:648] HybridClock initialized: now 1781894440034551 us; error 47 us; skew 500 ppm
I20260619 18:40:40.035087 4510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:40 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894440, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:40.039093 4510 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:40.040939 4510 webserver.cc:460] Webserver started at http://127.3.85.190:40049/ using document root <none> and password file <none>
I20260619 18:40:40.041791 4510 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:40.041898 4510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:40.042200 4510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:40.045459 4510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "2a0c6fe25f994c1d80293d02976c8853"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "d16962d8f19e57a7910f637251bfd2fc"
server_key_iv: "217cc7a4767c859e408eb859aab055e9"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.046267 4510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "2a0c6fe25f994c1d80293d02976c8853"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "d16962d8f19e57a7910f637251bfd2fc"
server_key_iv: "217cc7a4767c859e408eb859aab055e9"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.052440 4510 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:40:40.056615 4525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:40.058187 4510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:40.058368 4510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "2a0c6fe25f994c1d80293d02976c8853"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "d16962d8f19e57a7910f637251bfd2fc"
server_key_iv: "217cc7a4767c859e408eb859aab055e9"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.058643 4510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:40.076035 4510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:40.080690 4510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:40.080936 4510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:40.103989 4510 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:43591
I20260619 18:40:40.103951 4586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:43591 every 8 connection(s)
I20260619 18:40:40.105471 4510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:40.109561 4587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:40.114842 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4510
I20260619 18:40:40.115087 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:40.115715 3414 external_mini_cluster.cc:1428] Setting key fb4348f2dbb47d8dbb2549587b95f8d6
I20260619 18:40:40.121176 4587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Bootstrap starting.
I20260619 18:40:40.125180 4587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:40.126629 4587 log.cc:824] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:40.129854 4587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: No bootstrap required, opened a new log
Jun 19 18:40:40 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:40.138917 4587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } }
I20260619 18:40:40.139458 4587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:40.139588 4587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0c6fe25f994c1d80293d02976c8853, State: Initialized, Role: FOLLOWER
I20260619 18:40:40.140252 4587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } }
I20260619 18:40:40.140538 4587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:40.140695 4587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:40.140899 4587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:40.143347 4587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } }
I20260619 18:40:40.144057 4587 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0c6fe25f994c1d80293d02976c8853; no voters:
I20260619 18:40:40.144776 4587 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:40.144941 4592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:40.146235 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:40.117803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.3.85.190:43591)
0619 18:40:40.118718 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:40.118732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:40.118780 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:40:40.120511 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:40.120542 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:40.120970 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:40.121356 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:40.123146 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.124529 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.126416 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.127509 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.127767 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:40.137054 (+ 9287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:40.137064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:40.137088 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:40.137146 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:40.141017 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.141841 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.141851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.141861 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.142065 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.142509 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.142515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.142519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.143136 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:40.143457 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:40.143964 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 18:40:40.144470 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":404,"threads_started":1}
I20260619 18:40:40.147248 4587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:40.148329 4592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [term 1 LEADER]: Becoming Leader. State: Replica: 2a0c6fe25f994c1d80293d02976c8853, State: Running, Role: LEADER
I20260619 18:40:40.149034 4592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } }
I20260619 18:40:40.152513 4593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0c6fe25f994c1d80293d02976c8853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } } }
I20260619 18:40:40.152460 4594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0c6fe25f994c1d80293d02976c8853. Latest consensus state: current_term: 1 leader_uuid: "2a0c6fe25f994c1d80293d02976c8853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0c6fe25f994c1d80293d02976c8853" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43591 } } }
I20260619 18:40:40.153236 4594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:40.153083 4593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:40.155081 4603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:40.160650 4603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:40.172514 4603 catalog_manager.cc:1347] Generated new cluster ID: 7a229c04512647d8bdd61654277738ab
I20260619 18:40:40.172668 4603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:40.186085 4603 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:40.188217 4603 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:40.199679 4603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Generated new TSK 0
I20260619 18:40:40.200680 4603 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:40:40.248708 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43591
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:40.616279 4613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:40.616647 4613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:40.616822 4613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:40.624588 4613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:40.624771 4613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:40.624912 4613 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:40.625041 4613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:40:40.635372 4613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:40.636767 4613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:40.638267 4613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:40.652756 4618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:40.652806 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:40.652748 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:40.653705 4613 server_base.cc:1034] running on GCE node
I20260619 18:40:40.654500 4613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:40.655555 4613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:40.656826 4613 hybrid_clock.cc:648] HybridClock initialized: now 1781894440656739 us; error 70 us; skew 500 ppm
I20260619 18:40:40.657167 4613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:40 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894440, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:40.665934 4613 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:40:40.667754 4613 webserver.cc:460] Webserver started at http://127.3.85.129:33195/ using document root <none> and password file <none>
I20260619 18:40:40.668617 4613 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:40.668736 4613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:40.669144 4613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:40.672640 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "6fef01be43cf43a8acc175fd1bd3a4d7"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "35b5bad4d813215f730959d4a20a03b0"
server_key_iv: "4154c2c19073c5b3c7ad5bca2c42edec"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.673629 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "6fef01be43cf43a8acc175fd1bd3a4d7"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "35b5bad4d813215f730959d4a20a03b0"
server_key_iv: "4154c2c19073c5b3c7ad5bca2c42edec"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.680001 4613 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:40:40.684314 4628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:40.686012 4613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:40.686250 4613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "6fef01be43cf43a8acc175fd1bd3a4d7"
format_stamp: "Formatted at 2026-06-19 18:40:40 on dist-test-slave-09fp"
server_key: "35b5bad4d813215f730959d4a20a03b0"
server_key_iv: "4154c2c19073c5b3c7ad5bca2c42edec"
server_key_version: "encryptionkey@0"
I20260619 18:40:40.686542 4613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:40.711196 4613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:40.716395 4613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:40.716706 4613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:40.719734 4613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:40.719851 4613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:40.720013 4613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:40.720094 4613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:40 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894440, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:40.749044 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:40.720329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:58311 (local address 127.3.85.190:43591)
0619 18:40:40.727508 (+ 7179us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:40.727517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:40.727544 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:40:40.727616 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:40.727622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:40.727738 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:40.727884 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:40.730595 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.732113 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.735091 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.735892 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.736075 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:40.740737 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:40.740748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:40.740756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:40.740817 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:40.744959 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.746155 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.746162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.746167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.746355 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.747002 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.747007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.747013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.747384 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:40.747548 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:40.748417 (+ 869us) server_negotiation.cc:299] Negotiation successful
0619 18:40:40.748658 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6961,"thread_start_us":6841,"threads_started":1}
I20260619 18:40:40.750279 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:40.721150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.129:58311)
0619 18:40:40.722427 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:40.722525 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:40.726279 (+ 3754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:40.728191 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:40.728202 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:40.729161 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:40.730263 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:40.730303 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.732364 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.732368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:40.734882 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:40.734899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.736355 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.736364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:40.736604 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:40.737519 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:40.737550 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:40.740366 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:40.745156 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:40.745171 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:40.745196 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:40.745922 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:40.746542 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:40.746548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:40.746554 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:40.746832 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:40.747529 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:40.747538 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:40.748013 (+ 475us) client_negotiation.cc:769] Sending connection context
0619 18:40:40.748326 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 18:40:40.748939 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":482,"threads_started":1}
I20260619 18:40:40.774887 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:40.758009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.129:39869)
0619 18:40:40.758817 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:40.758834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:40.759019 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:40.759374 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:40.759378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:40.759959 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:40.760266 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:40.760275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.761273 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.761279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:40.762895 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:40.762921 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.764339 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.764344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:40.764478 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:40.765222 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:40.765258 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:40.766749 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:40.771309 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:40.771315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:40.771319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:40.771955 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:40.772484 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:40.772488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:40.772490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:40.772621 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:40.773322 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:40.773326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:40.773632 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 18:40:40.773863 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 18:40:40.774286 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":457,"threads_started":1}
I20260619 18:40:40.775239 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:40.757941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:39869 (local address 127.3.85.190:43591)
0619 18:40:40.758132 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:40.758137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:40.758931 (+ 794us) server_negotiation.cc:407] Connection header received
0619 18:40:40.759167 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:40.759172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:40.759237 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:40.759345 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:40.760421 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.761120 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.763148 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:40.764112 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:40.764307 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:40.767059 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:40.767070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:40.767078 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:40.767136 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:40.771147 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.772181 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.772187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.772190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.772359 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:40.772757 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:40.772761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:40.772765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:40.773176 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:40.773359 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:40.774687 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0619 18:40:40.774947 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:40:40.785352 4613 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:44571
I20260619 18:40:40.785388 4748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:44571 every 8 connection(s)
I20260619 18:40:40.786886 4613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:40.789328 4749 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43591
I20260619 18:40:40.789601 4749 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:40.790370 4749 heartbeater.cc:511] Master 127.3.85.190:43591 requested a full tablet report, sending...
I20260619 18:40:40.791316 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4613
I20260619 18:40:40.791534 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:40.792364 3414 external_mini_cluster.cc:1428] Setting key 1f9f90fef2390b75592373fe8820299a
I20260619 18:40:40.793066 4542 ts_manager.cc:194] Registered new tserver with Master: 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571)
I20260619 18:40:40.795002 4542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:58311
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:40:40.834298 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43591
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:41.180312 4753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:41.180631 4753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:41.180795 4753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:41.188397 4753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:41.188532 4753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:41.188686 4753 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:41.188818 4753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:40:41.199193 4753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:41.200621 4753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:41.202056 4753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:41.216557 4759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:41.216621 4761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:41.217149 4758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:41.217409 4753 server_base.cc:1034] running on GCE node
I20260619 18:40:41.218164 4753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:41.219314 4753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:41.220589 4753 hybrid_clock.cc:648] HybridClock initialized: now 1781894441220498 us; error 86 us; skew 500 ppm
I20260619 18:40:41.220898 4753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:41.224794 4753 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:40:41.226420 4753 webserver.cc:460] Webserver started at http://127.3.85.130:43069/ using document root <none> and password file <none>
I20260619 18:40:41.227286 4753 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:41.227414 4753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:41.227782 4753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:41.230983 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "7a682186fc19462d84a44b904b587284"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "538166abc2a6196347db804b9d27ab72"
server_key_iv: "8f0e26409da01cb8c8a45b4159c1f52f"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.231910 4753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "7a682186fc19462d84a44b904b587284"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "538166abc2a6196347db804b9d27ab72"
server_key_iv: "8f0e26409da01cb8c8a45b4159c1f52f"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.237776 4753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:40:41.242015 4768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:41.244024 4753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:40:41.244266 4753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "7a682186fc19462d84a44b904b587284"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "538166abc2a6196347db804b9d27ab72"
server_key_iv: "8f0e26409da01cb8c8a45b4159c1f52f"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.244560 4753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:41.263132 4753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:41.267926 4753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:41.268241 4753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:41.271116 4753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:41.271260 4753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:41.271415 4753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:41.271489 4753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:41.299291 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.272014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44749 (local address 127.3.85.190:43591)
0619 18:40:41.272230 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.272235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.274228 (+ 1993us) server_negotiation.cc:407] Connection header received
0619 18:40:41.276970 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.276981 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.277117 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.277252 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:41.280042 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.281295 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.283623 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.284088 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.284247 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.290589 (+ 6342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:41.290600 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:41.290606 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:41.290666 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:41.293825 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.295206 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.295214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.295219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.295393 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.296143 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.296148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.296152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.296572 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:41.296744 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.298688 (+ 1944us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.298933 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:40:41.299940 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.273036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.130:44749)
0619 18:40:41.274019 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.274071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.276423 (+ 2352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.277637 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.277656 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.278751 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:41.279746 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.279808 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.281543 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.281549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.283436 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.283454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.284370 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.284378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.284716 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.285672 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:41.285722 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:41.290162 (+ 4440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:41.294021 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.294034 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.294062 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.294977 (+ 915us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.295544 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.295550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.295554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.295980 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.296814 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:41.296822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:41.297460 (+ 638us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.297867 (+ 407us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.298362 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":467,"threads_started":1}
I20260619 18:40:41.322103 4753 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:38705
I20260619 18:40:41.322202 4887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:38705 every 8 connection(s)
I20260619 18:40:41.323797 4753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:41.324507 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4753
I20260619 18:40:41.324703 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:41.325278 3414 external_mini_cluster.cc:1428] Setting key 79ab4c81e88c33496df1aa61b70d8158
I20260619 18:40:41.325377 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.307378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.130:36113)
0619 18:40:41.308315 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.308344 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.308569 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.308942 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.308949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.309731 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:41.310237 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.310257 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.311843 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.311849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.313888 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.313918 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.315230 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.315242 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.315471 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.316350 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:41.316369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:41.317662 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:41.321891 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.321900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.321907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.322617 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.323073 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.323076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.323079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.323200 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.323942 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:41.323946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:41.324179 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.324370 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.324795 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":545,"threads_started":1}
I20260619 18:40:41.326339 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.307010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:36113 (local address 127.3.85.190:43591)
0619 18:40:41.307259 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.307267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.308469 (+ 1202us) server_negotiation.cc:407] Connection header received
0619 18:40:41.308705 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.308710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.308778 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.308867 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:41.310585 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.311711 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.314147 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.314976 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.315157 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.317917 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:41.317922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:41.317926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:41.317961 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:41.321692 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.322753 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.322760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.322764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.322937 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.323315 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.323321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.323324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.323781 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:41.323927 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.325709 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.325964 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:40:41.327728 4888 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43591
I20260619 18:40:41.328107 4888 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:41.329278 4888 heartbeater.cc:511] Master 127.3.85.190:43591 requested a full tablet report, sending...
I20260619 18:40:41.331250 4542 ts_manager.cc:194] Registered new tserver with Master: 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705)
I20260619 18:40:41.332161 4542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:44749
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:40:41.369868 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43591
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:41.718806 4892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:41.719103 4892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:41.719278 4892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:41.727100 4892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:41.727231 4892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:41.727372 4892 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:41.727492 4892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:40:41.737859 4892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:43591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:41.739269 4892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:41.740708 4892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:41.755775 4897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:41.755880 4898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:41.755825 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:41.756932 4892 server_base.cc:1034] running on GCE node
I20260619 18:40:41.757649 4892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:41.758996 4892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:41.760303 4892 hybrid_clock.cc:648] HybridClock initialized: now 1781894441760218 us; error 61 us; skew 500 ppm
I20260619 18:40:41.760643 4892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:41.764315 4892 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:40:41.765934 4892 webserver.cc:460] Webserver started at http://127.3.85.131:44737/ using document root <none> and password file <none>
I20260619 18:40:41.766817 4892 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:41.766961 4892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:41.767306 4892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:41.770514 4892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "ad6c561775ce42748a8bdfa3cfd92545"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "5d9d1fc8d41f66bc52e6f26251143b02"
server_key_iv: "25f22750da42a8fe2474f4cce8412aae"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.771442 4892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "ad6c561775ce42748a8bdfa3cfd92545"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "5d9d1fc8d41f66bc52e6f26251143b02"
server_key_iv: "25f22750da42a8fe2474f4cce8412aae"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.777006 4892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:40:41.781111 4907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:41.782687 4892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:40:41.782877 4892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "ad6c561775ce42748a8bdfa3cfd92545"
format_stamp: "Formatted at 2026-06-19 18:40:41 on dist-test-slave-09fp"
server_key: "5d9d1fc8d41f66bc52e6f26251143b02"
server_key_iv: "25f22750da42a8fe2474f4cce8412aae"
server_key_version: "encryptionkey@0"
I20260619 18:40:41.783128 4892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:41.798385 4749 heartbeater.cc:503] Master 127.3.85.190:43591 was elected leader, sending a full tablet report...
I20260619 18:40:41.809639 4892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:41.814980 4892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:41.815321 4892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:41.818531 4892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:41.818693 4892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 18:40:41.818898 4892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:41.819000 4892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:41.846128 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.819316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:35317 (local address 127.3.85.190:43591)
0619 18:40:41.819583 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.819590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.821215 (+ 1625us) server_negotiation.cc:407] Connection header received
0619 18:40:41.824438 (+ 3223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.824444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.824545 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.824673 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:41.827752 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.828523 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.830396 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.830966 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.831112 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.836809 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:41.836820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:41.836829 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:41.836890 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:41.841149 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.842572 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.842580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.842586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.842808 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.843512 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.843520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.843526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.844041 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:41.844257 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.845201 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.845559 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 18:40:41.846880 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.819975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.131:35317)
0619 18:40:41.820954 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.821026 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.823921 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.825193 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.825215 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.826244 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:41.827457 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.827493 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.828722 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.828727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.830209 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.830226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.831363 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.831369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.831673 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.832764 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:41.832811 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:41.836482 (+ 3671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:41.841430 (+ 4948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.841442 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.841467 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.842334 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.842956 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.842959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.842962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.843327 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.844258 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:41.844265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:41.844696 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.844971 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.845428 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":391,"threads_started":1}
I20260619 18:40:41.871240 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.854185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:46761 (local address 127.3.85.190:43591)
0619 18:40:41.854499 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.854510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.855137 (+ 627us) server_negotiation.cc:407] Connection header received
0619 18:40:41.855305 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.855315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.855456 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.855634 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:41.856620 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.857889 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.860438 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.861062 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.861217 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.863813 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:41.863819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:41.863823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:41.863863 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:41.867777 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.868861 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.868865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.868868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.869008 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.869477 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.869482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.869487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.869805 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:41.870231 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.870714 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.870963 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:40:41.872112 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.854203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.3.85.131:46761)
0619 18:40:41.855020 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.855039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.855173 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.855642 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.855645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.856132 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:41.856437 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.856447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.858142 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.858147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.860193 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.860217 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.861370 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.861379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.861572 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.862274 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:41.862293 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:41.863557 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:41.868018 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.868025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.868029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.868698 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.869135 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:41.869145 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:41.869149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:41.869348 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:41.869986 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:41.869998 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:41.870336 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.870613 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.871147 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":482,"threads_started":1}
I20260619 18:40:41.882038 4892 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46841
I20260619 18:40:41.882061 5026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46841 every 8 connection(s)
I20260619 18:40:41.883589 4892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:41.886102 5027 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43591
I20260619 18:40:41.886374 5027 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:41.887203 5027 heartbeater.cc:511] Master 127.3.85.190:43591 requested a full tablet report, sending...
I20260619 18:40:41.888866 4542 ts_manager.cc:194] Registered new tserver with Master: ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841)
I20260619 18:40:41.889671 4542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:35317
I20260619 18:40:41.891726 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 4892
I20260619 18:40:41.891920 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:41.892452 3414 external_mini_cluster.cc:1428] Setting key 77b735e2fe354c9678ccd8487b3e1128
I20260619 18:40:41.895720 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:40:41 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:41.944562 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.929553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.3.85.190:43591)
0619 18:40:41.929815 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.929823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.930104 (+ 281us) server_negotiation.cc:407] Connection header received
0619 18:40:41.930329 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.930335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.930410 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.930565 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:41.931997 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.932773 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.934102 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.934607 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.934753 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.938949 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:41.938955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:41.938959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:41.939101 (+ 142us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:41.941685 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.942439 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.942476 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.942480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.942641 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:41.943059 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:41.943065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:41.943069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:41.943364 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:41.943631 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.943940 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.944187 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 18:40:41.949451 4542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:41.957237 4542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:41.997360 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.970822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46841 (local address 127.0.0.1:40432)
0619 18:40:41.985824 (+ 15002us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.985877 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.986228 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.989738 (+ 3510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.989744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.989770 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:41.990286 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.990298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.992598 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.992604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.994400 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.994421 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.996203 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.996209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.996418 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.996477 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.996638 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.996862 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14697,"mutex_wait_us":4,"thread_start_us":17285,"threads_started":1}
I20260619 18:40:41.997560 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.972170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.3.85.129:44571)
0619 18:40:41.987078 (+ 14908us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.987092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.988529 (+ 1437us) server_negotiation.cc:407] Connection header received
0619 18:40:41.988709 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.988722 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.989048 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.989282 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:41.990022 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.991753 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.995040 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.996479 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.996628 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.996821 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.997072 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.997251 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14689,"thread_start_us":14534,"threads_started":1}
I20260619 18:40:41.997911 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.971615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44571 (local address 127.0.0.1:40500)
0619 18:40:41.988379 (+ 16764us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.988400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.988540 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.989293 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.989301 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.989332 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:41.989777 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.989793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.991876 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.991886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.994825 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.994841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.996711 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.996716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.996870 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.996904 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.997003 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.997146 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16617,"mutex_wait_us":192,"thread_start_us":15656,"threads_started":1}
I20260619 18:40:41.998131 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.970848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:38705 (local address 127.0.0.1:45966)
0619 18:40:41.987647 (+ 16799us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:41.987675 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:41.987832 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:41.989388 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:41.989398 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:41.989424 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:41.990069 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.990082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.991824 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.991832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.994826 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:41.994841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.996963 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.997039 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:41.997222 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.997258 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:40:41.997358 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:40:41.997492 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16617,"mutex_wait_us":224,"thread_start_us":14523,"threads_started":1}
I20260619 18:40:41.998422 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.971665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.3.85.130:38705)
0619 18:40:41.988737 (+ 17072us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.988748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.988776 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:40:41.988879 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.988887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.989163 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.990067 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:41.990270 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.991664 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.995045 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.996472 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.996624 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.996820 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.997913 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.998136 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16889,"thread_start_us":16798,"threads_started":1}
I20260619 18:40:41.998524 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:41.971090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.3.85.131:46841)
0619 18:40:41.989132 (+ 18042us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:41.989142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:41.989173 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:40:41.989265 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:41.989275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:41.989518 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:41.989720 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:41.990751 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.992441 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.994679 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:41.995988 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:41.996163 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:41.996420 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:41.997946 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0619 18:40:41.998147 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17872,"thread_start_us":17704,"threads_started":1}
I20260619 18:40:42.002097 4679 tablet_service.cc:1467] Processing CreateTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 (DEFAULT_TABLE table=test-table [id=edc4e4ce14514219840dd34f4c8a7619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.002104 4818 tablet_service.cc:1467] Processing CreateTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 (DEFAULT_TABLE table=test-table [id=edc4e4ce14514219840dd34f4c8a7619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.002077 4957 tablet_service.cc:1467] Processing CreateTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 (DEFAULT_TABLE table=test-table [id=edc4e4ce14514219840dd34f4c8a7619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.003571 4957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8afcceb960d4aa08339dd84e5a52a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.003609 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8afcceb960d4aa08339dd84e5a52a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.003602 4679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8afcceb960d4aa08339dd84e5a52a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.016121 5046 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Bootstrap starting.
I20260619 18:40:42.020390 5046 tablet_bootstrap.cc:654] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.021384 5047 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Bootstrap starting.
I20260619 18:40:42.021878 5046 log.cc:824] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:42.022367 5048 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Bootstrap starting.
I20260619 18:40:42.026120 5046 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: No bootstrap required, opened a new log
I20260619 18:40:42.026551 5046 ts_tablet_manager.cc:1397] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260619 18:40:42.027833 5048 tablet_bootstrap.cc:654] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.028108 5047 tablet_bootstrap.cc:654] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.029279 5048 log.cc:824] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:42.029409 5047 log.cc:824] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:42.032470 5048 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: No bootstrap required, opened a new log
I20260619 18:40:42.032809 5048 ts_tablet_manager.cc:1397] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 18:40:42.032821 5047 tablet_bootstrap.cc:492] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: No bootstrap required, opened a new log
I20260619 18:40:42.033118 5047 ts_tablet_manager.cc:1397] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260619 18:40:42.038178 5046 raft_consensus.cc:348] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.038760 5046 raft_consensus.cc:374] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.038865 5046 raft_consensus.cc:729] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6c561775ce42748a8bdfa3cfd92545, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.039598 5046 consensus_queue.cc:260] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.040854 5027 heartbeater.cc:503] Master 127.3.85.190:43591 was elected leader, sending a full tablet report...
I20260619 18:40:42.041600 5046 ts_tablet_manager.cc:1428] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 18:40:42.041440 5048 raft_consensus.cc:348] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.041895 5048 raft_consensus.cc:374] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.041986 5048 raft_consensus.cc:729] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a682186fc19462d84a44b904b587284, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.042760 5048 consensus_queue.cc:260] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.044159 4888 heartbeater.cc:503] Master 127.3.85.190:43591 was elected leader, sending a full tablet report...
I20260619 18:40:42.044704 5047 raft_consensus.cc:348] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.045224 5047 raft_consensus.cc:374] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.045336 5047 raft_consensus.cc:729] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef01be43cf43a8acc175fd1bd3a4d7, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.046073 5047 consensus_queue.cc:260] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.046272 5048 ts_tablet_manager.cc:1428] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 18:40:42.049489 5047 ts_tablet_manager.cc:1428] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
W20260619 18:40:42.082106 4889 tablet.cc:2369] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:42.135480 5028 tablet.cc:2369] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:42.164716 5053 raft_consensus.cc:482] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:42.165076 5053 raft_consensus.cc:504] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.166977 5053 leader_election.cc:288] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841), 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571)
I20260619 18:40:42.174305 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.167837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:56889 (local address 127.3.85.129:44571)
0619 18:40:42.168126 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.168135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.168897 (+ 762us) server_negotiation.cc:407] Connection header received
0619 18:40:42.169063 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.169068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.169158 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.169259 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.169898 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.170728 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.172520 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.173269 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.173477 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.173567 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.173912 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.174046 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 18:40:42.174525 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.167275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46841 (local address 127.3.85.130:60267)
0619 18:40:42.168520 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.168543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.168716 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.169079 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.169084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.169108 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.169485 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.169496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.170639 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.170645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.172265 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.172278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.173550 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.173556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.173738 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.173762 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.173869 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.174033 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":56,"thread_start_us":834,"threads_started":1}
I20260619 18:40:42.174592 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.167837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44571 (local address 127.3.85.130:56889)
0619 18:40:42.168738 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.168757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.168914 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.169286 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.169289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.169309 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.169731 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.169744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.170870 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.170874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.172330 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.172371 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.173516 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.173523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.173732 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.173757 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.173865 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.174051 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":84,"thread_start_us":506,"threads_started":1}
I20260619 18:40:42.174332 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.167326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:60267 (local address 127.3.85.131:46841)
0619 18:40:42.167561 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.167565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.168678 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 18:40:42.168866 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.168872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.168950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.169045 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.169648 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.170495 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.172451 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.173272 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.173482 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.173562 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.173901 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.174047 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:40:42.175738 4982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8afcceb960d4aa08339dd84e5a52a72" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6c561775ce42748a8bdfa3cfd92545" is_pre_election: true
I20260619 18:40:42.176007 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8afcceb960d4aa08339dd84e5a52a72" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" is_pre_election: true
I20260619 18:40:42.176308 4982 raft_consensus.cc:2454] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a682186fc19462d84a44b904b587284 in term 0.
I20260619 18:40:42.176537 4704 raft_consensus.cc:2454] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a682186fc19462d84a44b904b587284 in term 0.
I20260619 18:40:42.177122 4772 leader_election.cc:302] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a682186fc19462d84a44b904b587284, ad6c561775ce42748a8bdfa3cfd92545; no voters:
I20260619 18:40:42.177693 5053 raft_consensus.cc:2781] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:42.177868 5053 raft_consensus.cc:482] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:42.177984 5053 raft_consensus.cc:3037] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.180475 5053 raft_consensus.cc:504] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.181263 5053 leader_election.cc:288] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 election: Requested vote from peers ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841), 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571)
I20260619 18:40:42.181788 4982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8afcceb960d4aa08339dd84e5a52a72" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6c561775ce42748a8bdfa3cfd92545"
I20260619 18:40:42.181874 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8afcceb960d4aa08339dd84e5a52a72" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7"
I20260619 18:40:42.182051 4982 raft_consensus.cc:3037] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.182102 4704 raft_consensus.cc:3037] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.185397 4704 raft_consensus.cc:2454] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a682186fc19462d84a44b904b587284 in term 1.
I20260619 18:40:42.185397 4982 raft_consensus.cc:2454] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a682186fc19462d84a44b904b587284 in term 1.
I20260619 18:40:42.186002 4771 leader_election.cc:302] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fef01be43cf43a8acc175fd1bd3a4d7, 7a682186fc19462d84a44b904b587284; no voters:
I20260619 18:40:42.186524 5053 raft_consensus.cc:2781] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:42.187155 5053 raft_consensus.cc:686] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 1 LEADER]: Becoming Leader. State: Replica: 7a682186fc19462d84a44b904b587284, State: Running, Role: LEADER
I20260619 18:40:42.187716 5053 consensus_queue.cc:237] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } }
I20260619 18:40:42.192675 4542 catalog_manager.cc:5495] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a682186fc19462d84a44b904b587284 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "7a682186fc19462d84a44b904b587284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:42.246219 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.240867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.3.85.130:38705)
0619 18:40:42.241059 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.241064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.241170 (+ 106us) server_negotiation.cc:407] Connection header received
0619 18:40:42.241429 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.241433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.241511 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.241640 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:42.241964 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.242763 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.244114 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.244501 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.244603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.245149 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.245689 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.245801 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.245927 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.245995 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:42.255559 4704 raft_consensus.cc:1261] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Refusing update from remote peer 7a682186fc19462d84a44b904b587284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.255553 4982 raft_consensus.cc:1261] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Refusing update from remote peer 7a682186fc19462d84a44b904b587284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.256600 5057 consensus_queue.cc:1035] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.256976 5053 consensus_queue.cc:1035] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.269961 5060 mvcc.cc:205] Tried to move back new op lower bound from 7298639635468111872 to 7298639635205312512. Current Snapshot: MvccSnapshot[applied={T|T < 7298639635468111872}]
I20260619 18:40:42.295625 4541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:42.295943 4541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:42.299978 4541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Sending DeleteTablet for 3 replicas of tablet d8afcceb960d4aa08339dd84e5a52a72
I20260619 18:40:42.301671 4957 tablet_service.cc:1514] Processing DeleteTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40432
I20260619 18:40:42.301692 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45966
I20260619 18:40:42.302023 4679 tablet_service.cc:1514] Processing DeleteTablet for tablet d8afcceb960d4aa08339dd84e5a52a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40500
I20260619 18:40:42.306305 4541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:42.307379 4541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:42.317335 4956 tablet_service.cc:1467] Processing CreateTablet for tablet 325a9d1b29d642768b57c82b209d25bd (DEFAULT_TABLE table=test-table [id=ab7ae765a5944131a60207522c16991e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.317754 4678 tablet_service.cc:1467] Processing CreateTablet for tablet 325a9d1b29d642768b57c82b209d25bd (DEFAULT_TABLE table=test-table [id=ab7ae765a5944131a60207522c16991e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.318087 4956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325a9d1b29d642768b57c82b209d25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.318787 4678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325a9d1b29d642768b57c82b209d25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.321810 5070 tablet_replica.cc:330] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: stopping tablet replica
I20260619 18:40:42.322124 5072 tablet_replica.cc:330] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: stopping tablet replica
I20260619 18:40:42.322345 5070 raft_consensus.cc:2229] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:42.322541 5072 raft_consensus.cc:2229] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:42.322934 5070 raft_consensus.cc:2258] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.323143 5072 raft_consensus.cc:2258] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.324096 4817 tablet_service.cc:1467] Processing CreateTablet for tablet 325a9d1b29d642768b57c82b209d25bd (DEFAULT_TABLE table=test-table [id=ab7ae765a5944131a60207522c16991e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:42.325098 5070 ts_tablet_manager.cc:1905] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:42.325552 5072 ts_tablet_manager.cc:1905] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:42.327562 4817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325a9d1b29d642768b57c82b209d25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.329429 5047 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Bootstrap starting.
I20260619 18:40:42.331635 5047 tablet_bootstrap.cc:654] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.333300 5072 ts_tablet_manager.cc:1918] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:42.333463 5072 log.cc:1196] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/d8afcceb960d4aa08339dd84e5a52a72
I20260619 18:40:42.334120 5072 ts_tablet_manager.cc:1939] T d8afcceb960d4aa08339dd84e5a52a72 P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting consensus metadata
I20260619 18:40:42.338788 5046 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Bootstrap starting.
I20260619 18:40:42.340516 5046 tablet_bootstrap.cc:654] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.340893 4528 catalog_manager.cc:4841] TS 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571): tablet d8afcceb960d4aa08339dd84e5a52a72 (table test-table [id=edc4e4ce14514219840dd34f4c8a7619]) successfully deleted
I20260619 18:40:42.342684 5070 ts_tablet_manager.cc:1918] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:42.342932 5070 log.cc:1196] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/d8afcceb960d4aa08339dd84e5a52a72
I20260619 18:40:42.343102 5047 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: No bootstrap required, opened a new log
I20260619 18:40:42.343261 5047 ts_tablet_manager.cc:1397] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260619 18:40:42.344111 5070 ts_tablet_manager.cc:1939] T d8afcceb960d4aa08339dd84e5a52a72 P ad6c561775ce42748a8bdfa3cfd92545: Deleting consensus metadata
I20260619 18:40:42.344128 5047 raft_consensus.cc:348] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.344485 5047 raft_consensus.cc:374] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.344611 5047 raft_consensus.cc:729] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef01be43cf43a8acc175fd1bd3a4d7, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.344965 5047 consensus_queue.cc:260] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.346534 4529 catalog_manager.cc:4841] TS ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841): tablet d8afcceb960d4aa08339dd84e5a52a72 (table test-table [id=edc4e4ce14514219840dd34f4c8a7619]) successfully deleted
I20260619 18:40:42.348116 5047 ts_tablet_manager.cc:1428] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 18:40:42.350605 5048 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Bootstrap starting.
I20260619 18:40:42.352240 5054 raft_consensus.cc:482] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:42.352541 5048 tablet_bootstrap.cc:654] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.352445 5054 raft_consensus.cc:504] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.354707 5054 leader_election.cc:288] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705), ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841)
I20260619 18:40:42.355847 5046 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: No bootstrap required, opened a new log
I20260619 18:40:42.356207 5046 ts_tablet_manager.cc:1397] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260619 18:40:42.357518 5046 raft_consensus.cc:348] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.357918 5046 raft_consensus.cc:374] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.358052 5046 raft_consensus.cc:729] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6c561775ce42748a8bdfa3cfd92545, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.358549 5046 consensus_queue.cc:260] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.363533 5046 ts_tablet_manager.cc:1428] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260619 18:40:42.364657 5071 tablet_replica.cc:330] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: stopping tablet replica
I20260619 18:40:42.366084 5071 raft_consensus.cc:2229] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:42.367870 5071 raft_consensus.cc:2258] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.376335 5071 ts_tablet_manager.cc:1905] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:42.377915 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.355380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:56835 (local address 127.3.85.130:38705)
0619 18:40:42.355545 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.355551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.368135 (+ 12584us) server_negotiation.cc:407] Connection header received
0619 18:40:42.368497 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.368503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.368579 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.368661 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.370721 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.371994 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.374994 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.376378 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.376543 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.376633 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.377390 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.377568 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:40:42.378849 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.355298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:35985 (local address 127.3.85.131:46841)
0619 18:40:42.355478 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.355485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.370780 (+ 15295us) server_negotiation.cc:407] Connection header received
0619 18:40:42.371051 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.371058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.371145 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.371212 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.371986 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.373177 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.375875 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.377194 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.377347 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.377429 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.378398 (+ 969us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.378577 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:40:42.378899 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.354918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:38705 (local address 127.3.85.129:56835)
0619 18:40:42.367949 (+ 13031us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.367983 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.368304 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.368940 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.368948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.368991 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.369498 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.369515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.372186 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.372191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.374755 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.374775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.376900 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.376909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.377134 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.377177 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.378225 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.378429 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12823,"mutex_wait_us":145,"thread_start_us":15393,"threads_started":1}
I20260619 18:40:42.379729 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.355199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46841 (local address 127.3.85.129:35985)
0619 18:40:42.370679 (+ 15480us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.370700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.370891 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.371442 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.371447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.371473 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.371803 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.371815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.373365 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.373370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.375635 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.375651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.377643 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.377652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.377873 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.377923 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.379136 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.379306 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15329,"mutex_wait_us":1,"thread_start_us":12870,"threads_started":1}
I20260619 18:40:42.379941 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a682186fc19462d84a44b904b587284" is_pre_election: true
I20260619 18:40:42.380462 4982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6c561775ce42748a8bdfa3cfd92545" is_pre_election: true
I20260619 18:40:42.381134 4982 raft_consensus.cc:2454] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fef01be43cf43a8acc175fd1bd3a4d7 in term 0.
W20260619 18:40:42.381835 4629 leader_election.cc:341] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705): Illegal state: must be running to vote when last-logged opid is not known
I20260619 18:40:42.382217 4632 leader_election.cc:302] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fef01be43cf43a8acc175fd1bd3a4d7, ad6c561775ce42748a8bdfa3cfd92545; no voters: 7a682186fc19462d84a44b904b587284
I20260619 18:40:42.382735 5054 raft_consensus.cc:2781] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:42.382845 5054 raft_consensus.cc:482] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:42.382934 5054 raft_consensus.cc:3037] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.387162 5054 raft_consensus.cc:504] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.388657 4982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6c561775ce42748a8bdfa3cfd92545"
I20260619 18:40:42.388756 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a682186fc19462d84a44b904b587284"
I20260619 18:40:42.388900 4982 raft_consensus.cc:3037] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Advancing to term 1
W20260619 18:40:42.389477 4629 leader_election.cc:341] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705): Illegal state: must be running to vote when last-logged opid is not known
I20260619 18:40:42.390514 5048 tablet_bootstrap.cc:492] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: No bootstrap required, opened a new log
I20260619 18:40:42.390760 5048 ts_tablet_manager.cc:1397] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.008s
I20260619 18:40:42.391775 5048 raft_consensus.cc:348] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.391952 4982 raft_consensus.cc:2454] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fef01be43cf43a8acc175fd1bd3a4d7 in term 1.
I20260619 18:40:42.392031 5048 raft_consensus.cc:374] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.392131 5048 raft_consensus.cc:729] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a682186fc19462d84a44b904b587284, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.392428 5048 consensus_queue.cc:260] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.392721 4632 leader_election.cc:302] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6fef01be43cf43a8acc175fd1bd3a4d7, ad6c561775ce42748a8bdfa3cfd92545; no voters: 7a682186fc19462d84a44b904b587284
I20260619 18:40:42.393062 5048 ts_tablet_manager.cc:1428] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 18:40:42.393369 5054 leader_election.cc:288] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705), ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841)
I20260619 18:40:42.393600 5054 raft_consensus.cc:2781] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:42.393741 5054 raft_consensus.cc:686] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6fef01be43cf43a8acc175fd1bd3a4d7, State: Running, Role: LEADER
I20260619 18:40:42.394666 5054 consensus_queue.cc:237] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.399595 4541 catalog_manager.cc:5495] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:42.399788 5071 ts_tablet_manager.cc:1918] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:42.400035 5071 log.cc:1196] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/d8afcceb960d4aa08339dd84e5a52a72
I20260619 18:40:42.400667 5071 ts_tablet_manager.cc:1939] T d8afcceb960d4aa08339dd84e5a52a72 P 7a682186fc19462d84a44b904b587284: Deleting consensus metadata
I20260619 18:40:42.402691 4526 catalog_manager.cc:4841] TS 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705): tablet d8afcceb960d4aa08339dd84e5a52a72 (table test-table [id=edc4e4ce14514219840dd34f4c8a7619]) successfully deleted
Jun 19 18:40:42 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894440, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:42.463161 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.445557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43591 (local address 127.0.0.1:47660)
0619 18:40:42.446733 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.446751 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.446945 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.447318 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.447322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.448114 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:42.448345 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.448356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.449808 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.449811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.451166 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.451188 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.451894 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.451899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.452184 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.453055 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:42.453097 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:42.456009 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:42.459800 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:42.459820 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:42.459829 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:42.460489 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:42.460953 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:42.460958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:42.460961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:42.461144 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:42.461693 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:42.461702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:42.461927 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.462143 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.462542 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":53,"thread_start_us":492,"threads_started":1}
I20260619 18:40:42.463415 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.445559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.3.85.190:43591)
0619 18:40:42.446106 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.446117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.446850 (+ 733us) server_negotiation.cc:407] Connection header received
0619 18:40:42.447094 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.447104 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.447173 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.447288 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:42.448557 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.449628 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.451325 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.451733 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.451885 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.456286 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:42.456292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:42.456295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:42.456328 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:42.459610 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:42.460671 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:42.460676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:42.460678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:42.460827 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:42.461260 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:42.461263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:42.461265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:42.461555 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:42.461673 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.462293 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.462682 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":395,"thread_start_us":305,"threads_started":1}
I20260619 18:40:42.469370 4541 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:47660:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:40:42.470795 4541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:42.478665 4678 tablet_service.cc:1467] Processing CreateTablet for tablet 74178cd13ac74a7584551383739c9ec1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=508cf79f1d8946d18bcc715e36ad62e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:42.479207 4678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74178cd13ac74a7584551383739c9ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.479413 4956 tablet_service.cc:1467] Processing CreateTablet for tablet 74178cd13ac74a7584551383739c9ec1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=508cf79f1d8946d18bcc715e36ad62e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:42.479967 4956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74178cd13ac74a7584551383739c9ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.483665 4818 tablet_service.cc:1467] Processing CreateTablet for tablet 74178cd13ac74a7584551383739c9ec1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=508cf79f1d8946d18bcc715e36ad62e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:42.484269 4818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74178cd13ac74a7584551383739c9ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:42.485435 5047 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: Bootstrap starting.
I20260619 18:40:42.487325 5047 tablet_bootstrap.cc:654] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.487562 5046 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: Bootstrap starting.
I20260619 18:40:42.489809 5046 tablet_bootstrap.cc:654] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.492857 5048 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: Bootstrap starting.
I20260619 18:40:42.495180 5048 tablet_bootstrap.cc:654] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:42.500155 5048 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: No bootstrap required, opened a new log
I20260619 18:40:42.500375 5048 ts_tablet_manager.cc:1397] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 18:40:42.501281 5048 raft_consensus.cc:348] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.501550 5048 raft_consensus.cc:374] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.501637 5048 raft_consensus.cc:729] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a682186fc19462d84a44b904b587284, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.501672 5046 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: No bootstrap required, opened a new log
I20260619 18:40:42.501731 5047 tablet_bootstrap.cc:492] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: No bootstrap required, opened a new log
I20260619 18:40:42.501909 5046 ts_tablet_manager.cc:1397] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.001s
I20260619 18:40:42.501909 5047 ts_tablet_manager.cc:1397] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260619 18:40:42.501936 5048 consensus_queue.cc:260] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.502599 5048 ts_tablet_manager.cc:1428] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 18:40:42.502768 5047 raft_consensus.cc:348] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.503039 5047 raft_consensus.cc:374] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.503141 5047 raft_consensus.cc:729] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef01be43cf43a8acc175fd1bd3a4d7, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.503881 5047 consensus_queue.cc:260] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.504408 5054 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.504720 5054 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.505263 5047 ts_tablet_manager.cc:1428] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:42.505293 5046 raft_consensus.cc:348] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.505600 5046 raft_consensus.cc:374] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:42.505721 5046 raft_consensus.cc:729] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6c561775ce42748a8bdfa3cfd92545, State: Initialized, Role: FOLLOWER
I20260619 18:40:42.505707 5065 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.506158 5065 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.506070 5046 consensus_queue.cc:260] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.506605 5052 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.506950 5052 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.507920 5046 ts_tablet_manager.cc:1428] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20260619 18:40:42.540972 4750 tablet.cc:2369] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:42.600872 5052 raft_consensus.cc:482] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:42.601126 5052 raft_consensus.cc:504] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.603106 5052 leader_election.cc:288] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571), 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705)
I20260619 18:40:42.615907 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.603720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:55915 (local address 127.3.85.129:44571)
0619 18:40:42.603906 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.603912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.607502 (+ 3590us) server_negotiation.cc:407] Connection header received
0619 18:40:42.607699 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.607709 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.607829 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.607973 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.608792 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.609957 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.613047 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.614800 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.615047 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.615148 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.615429 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.615597 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:40:42.616365 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.604041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:44255 (local address 127.3.85.130:38705)
0619 18:40:42.604201 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.604207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.608263 (+ 4056us) server_negotiation.cc:407] Connection header received
0619 18:40:42.608472 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.608480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.608570 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.608697 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:42.609287 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.610427 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.613577 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.615203 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.615462 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.615547 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.615879 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.616030 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:42.616906 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.603503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:38705 (local address 127.3.85.131:44255)
0619 18:40:42.608126 (+ 4623us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.608146 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.608301 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.608722 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.608728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.608755 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.609092 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.609104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.610587 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.610593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.613308 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.613336 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.615486 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.615493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.615691 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.615716 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.615849 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.616248 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4487,"mutex_wait_us":246,"spinlock_wait_cycles":896,"thread_start_us":3983,"threads_started":1}
I20260619 18:40:42.617365 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.603503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44571 (local address 127.3.85.131:55915)
0619 18:40:42.607321 (+ 3818us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.607345 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.607529 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.608101 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.608113 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.608147 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:42.608567 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.608583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.610168 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.610174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.612587 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.612687 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.615030 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.615036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.615225 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.615260 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.616195 (+ 935us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.616967 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3656,"spinlock_wait_cycles":38656,"thread_start_us":3479,"threads_started":1}
I20260619 18:40:42.617797 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74178cd13ac74a7584551383739c9ec1" candidate_uuid: "ad6c561775ce42748a8bdfa3cfd92545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a682186fc19462d84a44b904b587284" is_pre_election: true
I20260619 18:40:42.618079 4838 raft_consensus.cc:2454] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6c561775ce42748a8bdfa3cfd92545 in term 0.
I20260619 18:40:42.618305 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74178cd13ac74a7584551383739c9ec1" candidate_uuid: "ad6c561775ce42748a8bdfa3cfd92545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" is_pre_election: true
I20260619 18:40:42.618629 4704 raft_consensus.cc:2454] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6c561775ce42748a8bdfa3cfd92545 in term 0.
I20260619 18:40:42.618757 4908 leader_election.cc:302] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a682186fc19462d84a44b904b587284, ad6c561775ce42748a8bdfa3cfd92545; no voters:
I20260619 18:40:42.619244 5052 raft_consensus.cc:2781] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:42.619365 5052 raft_consensus.cc:482] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:42.619446 5052 raft_consensus.cc:3037] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.622689 5052 raft_consensus.cc:504] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.624133 5052 leader_election.cc:288] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [CANDIDATE]: Term 1 election: Requested vote from peers 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571), 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705)
I20260619 18:40:42.624552 4838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74178cd13ac74a7584551383739c9ec1" candidate_uuid: "ad6c561775ce42748a8bdfa3cfd92545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a682186fc19462d84a44b904b587284"
I20260619 18:40:42.624784 4838 raft_consensus.cc:3037] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.626716 4704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74178cd13ac74a7584551383739c9ec1" candidate_uuid: "ad6c561775ce42748a8bdfa3cfd92545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7"
I20260619 18:40:42.626956 4704 raft_consensus.cc:3037] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.628381 4838 raft_consensus.cc:2454] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6c561775ce42748a8bdfa3cfd92545 in term 1.
I20260619 18:40:42.629007 4908 leader_election.cc:302] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a682186fc19462d84a44b904b587284, ad6c561775ce42748a8bdfa3cfd92545; no voters:
I20260619 18:40:42.629382 5052 raft_consensus.cc:2781] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:42.630059 4704 raft_consensus.cc:2454] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6c561775ce42748a8bdfa3cfd92545 in term 1.
I20260619 18:40:42.630697 5052 raft_consensus.cc:686] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [term 1 LEADER]: Becoming Leader. State: Replica: ad6c561775ce42748a8bdfa3cfd92545, State: Running, Role: LEADER
I20260619 18:40:42.631127 5052 consensus_queue.cc:237] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.634197 5092 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: TxnStatusTablet state changed. Reason: New leader ad6c561775ce42748a8bdfa3cfd92545. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.634619 5092 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:42.635375 5094 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:40:42.635637 5094 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:40:42.635815 4540 catalog_manager.cc:5495] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131). New cstate: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } health_report { overall_health: HEALTHY } } }
I20260619 18:40:42.665776 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.657662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.3.85.131:46841)
0619 18:40:42.658316 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.658322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.658340 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:40:42.658487 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.658495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.658624 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.658775 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:42.659262 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.660431 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.662488 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.663302 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.664079 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.664224 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.664993 (+ 769us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.665149 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.665334 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.665436 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260619 18:40:42.665791 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.657562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46841 (local address 127.0.0.1:40438)
0619 18:40:42.657808 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.657842 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.658011 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.658835 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.658841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.658856 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:42.659079 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.659092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.660600 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.660605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.662297 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.662317 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.663551 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.663557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.663765 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.663859 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.665143 (+ 1284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.665171 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.665293 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.665498 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:40:42.671214 4704 raft_consensus.cc:1261] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Refusing update from remote peer ad6c561775ce42748a8bdfa3cfd92545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.671301 4838 raft_consensus.cc:1261] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Refusing update from remote peer ad6c561775ce42748a8bdfa3cfd92545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.672053 5052 consensus_queue.cc:1035] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.672780 5092 consensus_queue.cc:1035] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.675698 5057 raft_consensus.cc:482] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:42.675884 5057 raft_consensus.cc:504] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } }
I20260619 18:40:42.676575 5057 leader_election.cc:288] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571), ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841)
I20260619 18:40:42.677821 4703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" is_pre_election: true
I20260619 18:40:42.678560 4982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325a9d1b29d642768b57c82b209d25bd" candidate_uuid: "7a682186fc19462d84a44b904b587284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6c561775ce42748a8bdfa3cfd92545" is_pre_election: true
I20260619 18:40:42.678902 4982 raft_consensus.cc:2379] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a682186fc19462d84a44b904b587284 in current term 1: Already voted for candidate 6fef01be43cf43a8acc175fd1bd3a4d7 in this term.
I20260619 18:40:42.679903 4771 leader_election.cc:302] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a682186fc19462d84a44b904b587284; no voters: 6fef01be43cf43a8acc175fd1bd3a4d7, ad6c561775ce42748a8bdfa3cfd92545
I20260619 18:40:42.680469 5057 raft_consensus.cc:3037] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:42.685024 5057 raft_consensus.cc:2726] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:40:42.689307 5065 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: TxnStatusTablet state changed. Reason: New leader ad6c561775ce42748a8bdfa3cfd92545. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.689694 5065 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.692248 5092 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.692561 5092 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:42.692791 5062 mvcc.cc:205] Tried to move back new op lower bound from 7298639637172506624 to 7298639637021380608. Current Snapshot: MvccSnapshot[applied={T|T < 7298639637172506624}]
I20260619 18:40:42.702020 5065 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.702345 5065 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 7a682186fc19462d84a44b904b587284: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.703598 5054 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: TxnStatusTablet state changed. Reason: New leader ad6c561775ce42748a8bdfa3cfd92545. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.703940 5054 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.714730 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.706806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.3.85.129:44571)
0619 18:40:42.706994 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.707000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.707021 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:40:42.707136 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.707142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.707268 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.707467 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:42.708127 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.708965 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.711064 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.711759 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.711939 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.713032 (+ 1093us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.713844 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.714001 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.714272 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.714379 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:40:42.719630 5052 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.720009 5052 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P ad6c561775ce42748a8bdfa3cfd92545: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:42.721005 5061 mvcc.cc:205] Tried to move back new op lower bound from 7298639637172506624 to 7298639637021380608. Current Snapshot: MvccSnapshot[applied={T|T < 7298639637172506624}]
I20260619 18:40:42.721979 5054 tablet_replica.cc:439] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad6c561775ce42748a8bdfa3cfd92545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef01be43cf43a8acc175fd1bd3a4d7" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44571 } } peers { permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 } } peers { permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 } } }
I20260619 18:40:42.722280 5054 tablet_replica.cc:442] T 74178cd13ac74a7584551383739c9ec1 P 6fef01be43cf43a8acc175fd1bd3a4d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:42.746757 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.734226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:57047 (local address 127.3.85.131:46841)
0619 18:40:42.734501 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.734507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.739261 (+ 4754us) server_negotiation.cc:407] Connection header received
0619 18:40:42.739482 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.739487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.739599 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.739729 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:42.740289 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.741898 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.744135 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.744990 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.745145 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.745797 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.745981 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.746085 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.746260 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.746350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:40:42.746878 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.734098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46841 (local address 127.3.85.129:57047)
0619 18:40:42.739096 (+ 4998us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.739120 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.739324 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.739762 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.739767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.739783 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:42.740102 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.740114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.742081 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.742088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.743919 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.743941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.745222 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.745229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.745478 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.745565 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.746105 (+ 540us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.746118 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.746208 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.746387 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4767,"thread_start_us":4584,"threads_started":1}
I20260619 18:40:42.759743 4838 raft_consensus.cc:1261] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Refusing update from remote peer 6fef01be43cf43a8acc175fd1bd3a4d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.759718 4982 raft_consensus.cc:1261] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Refusing update from remote peer 6fef01be43cf43a8acc175fd1bd3a4d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:42.760421 5078 consensus_queue.cc:1035] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a682186fc19462d84a44b904b587284" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.761497 5054 consensus_queue.cc:1035] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6c561775ce42748a8bdfa3cfd92545" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:42.862020 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.853209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:41529 (local address 127.3.85.129:44571)
0619 18:40:42.853468 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.853475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.854283 (+ 808us) server_negotiation.cc:407] Connection header received
0619 18:40:42.854678 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.854685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.854785 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.854857 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:42.855676 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.856777 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.859368 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.860103 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.860223 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.860996 (+ 773us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.861164 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.861236 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:42.861557 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 18:40:42.861658 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 18:40:42.862993 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.852672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44571 (local address 127.3.85.131:41529)
0619 18:40:42.854149 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:42.854184 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:42.854483 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:42.855065 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:42.855070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:42.855084 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:42.855496 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.855510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.856984 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.856990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.858905 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:42.858928 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.860467 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.860475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:42.860677 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.860764 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:42.861337 (+ 573us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:42.861411 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 18:40:42.862323 (+ 912us) client_negotiation.cc:240] Negotiation successful
0619 18:40:42.862537 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":3370,"threads_started":1}
I20260619 18:40:42.942320 4540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:42.942662 4540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:42.946925 4540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a0c6fe25f994c1d80293d02976c8853: Sending DeleteTablet for 3 replicas of tablet 325a9d1b29d642768b57c82b209d25bd
I20260619 18:40:42.949371 4818 tablet_service.cc:1514] Processing DeleteTablet for tablet 325a9d1b29d642768b57c82b209d25bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45966
I20260619 18:40:42.949651 4678 tablet_service.cc:1514] Processing DeleteTablet for tablet 325a9d1b29d642768b57c82b209d25bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40500
I20260619 18:40:42.950158 4956 tablet_service.cc:1514] Processing DeleteTablet for tablet 325a9d1b29d642768b57c82b209d25bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:42 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40432
I20260619 18:40:42.961642 5108 tablet_replica.cc:330] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: stopping tablet replica
I20260619 18:40:42.961930 5108 raft_consensus.cc:2229] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:42.962216 5108 raft_consensus.cc:2258] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.965557 5108 ts_tablet_manager.cc:1905] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:42.979496 5111 tablet_replica.cc:330] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: stopping tablet replica
I20260619 18:40:42.980181 5109 tablet_replica.cc:330] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: stopping tablet replica
I20260619 18:40:42.980528 5111 raft_consensus.cc:2229] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:42.980865 5109 raft_consensus.cc:2229] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:42.981233 5111 raft_consensus.cc:2258] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.981870 5109 raft_consensus.cc:2258] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:42.983278 5111 ts_tablet_manager.cc:1905] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:42.983942 5108 ts_tablet_manager.cc:1918] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:42.984094 5108 log.cc:1196] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/325a9d1b29d642768b57c82b209d25bd
I20260619 18:40:42.984563 5108 ts_tablet_manager.cc:1939] T 325a9d1b29d642768b57c82b209d25bd P 7a682186fc19462d84a44b904b587284: Deleting consensus metadata
I20260619 18:40:42.985819 4526 catalog_manager.cc:4841] TS 7a682186fc19462d84a44b904b587284 (127.3.85.130:38705): tablet 325a9d1b29d642768b57c82b209d25bd (table test-table [id=ab7ae765a5944131a60207522c16991e]) successfully deleted
I20260619 18:40:42.986701 5109 ts_tablet_manager.cc:1905] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:40:42 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:43.001188 5111 ts_tablet_manager.cc:1918] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:43.004539 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:42.960262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.3.85.129:44571)
0619 18:40:42.960506 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:42.960514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:42.976453 (+ 15939us) server_negotiation.cc:407] Connection header received
0619 18:40:42.977102 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:42.977112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:42.977218 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:42.977306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:42.985726 (+ 8420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.986981 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.989488 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:42.990164 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:42.990267 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:42.995796 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:42.995811 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:42.995831 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:42.995888 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:43.000162 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.001294 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.001304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.001308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.001511 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.002018 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.002024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.002028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.002609 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:43.002947 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:43.003609 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 18:40:43.003879 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:40:43.006031 5111 log.cc:1196] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/325a9d1b29d642768b57c82b209d25bd
W20260619 18:40:43.006223 4744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40518
I20260619 18:40:43.006611 5111 ts_tablet_manager.cc:1939] T 325a9d1b29d642768b57c82b209d25bd P ad6c561775ce42748a8bdfa3cfd92545: Deleting consensus metadata
I20260619 18:40:43.007853 4529 catalog_manager.cc:4841] TS ad6c561775ce42748a8bdfa3cfd92545 (127.3.85.131:46841): tablet 325a9d1b29d642768b57c82b209d25bd (table test-table [id=ab7ae765a5944131a60207522c16991e]) successfully deleted
I20260619 18:40:43.008692 5109 ts_tablet_manager.cc:1918] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:43.008889 5109 log.cc:1196] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/325a9d1b29d642768b57c82b209d25bd
I20260619 18:40:43.009428 5109 ts_tablet_manager.cc:1939] T 325a9d1b29d642768b57c82b209d25bd P 6fef01be43cf43a8acc175fd1bd3a4d7: Deleting consensus metadata
I20260619 18:40:43.010607 4528 catalog_manager.cc:4841] TS 6fef01be43cf43a8acc175fd1bd3a4d7 (127.3.85.129:44571): tablet 325a9d1b29d642768b57c82b209d25bd (table test-table [id=ab7ae765a5944131a60207522c16991e]) successfully deleted
I20260619 18:40:43.045064 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:43.012132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.3.85.190:43591)
0619 18:40:43.024706 (+ 12574us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:43.024713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:43.025395 (+ 682us) server_negotiation.cc:407] Connection header received
0619 18:40:43.025617 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:43.025623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:43.025712 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:43.025850 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:43.028008 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.029652 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.031831 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.032558 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.032714 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:43.036412 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:43.036419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:43.036424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:43.036499 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:43.040613 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.041715 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.041722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.041726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.041906 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.042404 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.042410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.042414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.043286 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:43.043581 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:43.043978 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 18:40:43.044245 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12410,"thread_start_us":12590,"threads_started":1}
W20260619 18:40:43.045830 4540 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47664
I20260619 18:40:43.048419 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4613
I20260619 18:40:43.069491 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:43.050908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.3.85.129:44571)
0619 18:40:43.051114 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:43.051120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:43.051590 (+ 470us) server_negotiation.cc:407] Connection header received
0619 18:40:43.051893 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:43.051900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:43.052003 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:43.052085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:43.053914 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.055254 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.057185 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.057920 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.058070 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:43.061265 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:43.061274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:43.061279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:43.061340 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:43.065613 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.066666 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.066673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.066677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.066887 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.067351 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.067357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.067360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.067771 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:43.068060 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:43.068408 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:40:43.068684 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:40:43.070272 4744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40520
W20260619 18:40:43.070987 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4613: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:43.071202 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4613
I20260619 18:40:43.087155 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4753
Jun 19 18:40:43 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:40:43.103853 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:43.088085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.3.85.130:38705)
0619 18:40:43.088293 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:43.088299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:43.088325 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:40:43.088514 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:43.088520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:43.088597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:43.088706 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:43.089897 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.091184 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.092479 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.092933 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.093077 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:43.097611 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:43.097626 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:43.097643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:43.097688 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:43.100737 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.101563 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.101580 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.101583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.101724 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.102124 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.102130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.102133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.102611 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:43.102938 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:43.103262 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 18:40:43.103524 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:40:43.104590 4883 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45980
W20260619 18:40:43.105237 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4753: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:43.105427 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4753
I20260619 18:40:43.132208 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4892
Jun 19 18:40:43 dist-test-slave-09fp krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:40:43.149072 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:43.132975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.3.85.131:46841)
0619 18:40:43.133166 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:43.133174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:43.133330 (+ 156us) server_negotiation.cc:407] Connection header received
0619 18:40:43.133547 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:43.133554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:43.133621 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:43.133749 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:43.135107 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.135834 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.137057 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:43.137547 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:43.137700 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:43.142506 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:43.142532 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:43.142549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:43.142598 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:43.145779 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.146857 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.146867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.146871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.147021 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:43.147499 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:43.147502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:43.147504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:43.147878 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:43.148109 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:43.148547 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 18:40:43.148733 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 18:40:43.149946 5022 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40446
W20260619 18:40:43.150753 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4892: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:43.150925 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4892
I20260619 18:40:43.172505 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 4510
I20260619 18:40:43.173802 4582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:43.289255 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 4510
2026-06-19T18:40:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4930 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:43 dist-test-slave-09fp krb5kdc[5120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:43 dist-test-slave-09fp krb5kdc[5120](info): set up 2 sockets
Jun 19 18:40:43 dist-test-slave-09fp krb5kdc[5120](info): commencing operation
krb5kdc: starting...
W20260619 18:40:44.386878 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:44 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:44.590335 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39101
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:44.938624 5133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:44.938938 5133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:44.939110 5133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:44.946537 5133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:44.946660 5133 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:40:44.946755 5133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:44.946874 5133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:44.946933 5133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:44.947012 5133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:44.957366 5133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39101
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:44.958914 5133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:44.960312 5133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:44.972988 5139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:44.972990 5141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:44.972958 5138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:44.974272 5133 server_base.cc:1034] running on GCE node
I20260619 18:40:44.975215 5133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:44.976822 5133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:44.978132 5133 hybrid_clock.cc:648] HybridClock initialized: now 1781894444978065 us; error 40 us; skew 500 ppm
I20260619 18:40:44.978591 5133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:44 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894444, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:44.982385 5133 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:44.984284 5133 webserver.cc:460] Webserver started at http://127.3.85.190:36081/ using document root <none> and password file <none>
I20260619 18:40:44.985121 5133 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:44.985280 5133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:44.985625 5133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:44.988965 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "e4c231c9a7c544c8935cf3f23796be91"
format_stamp: "Formatted at 2026-06-19 18:40:44 on dist-test-slave-09fp"
server_key: "04997180282faba1a27f4a43ed27af27"
server_key_iv: "af6c68fbbf9d7053616c2b1d6d2684fc"
server_key_version: "encryptionkey@0"
I20260619 18:40:44.989879 5133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "e4c231c9a7c544c8935cf3f23796be91"
format_stamp: "Formatted at 2026-06-19 18:40:44 on dist-test-slave-09fp"
server_key: "04997180282faba1a27f4a43ed27af27"
server_key_iv: "af6c68fbbf9d7053616c2b1d6d2684fc"
server_key_version: "encryptionkey@0"
I20260619 18:40:44.995940 5133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:40:45.000166 5148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:45.001843 5133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:40:45.002101 5133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "e4c231c9a7c544c8935cf3f23796be91"
format_stamp: "Formatted at 2026-06-19 18:40:44 on dist-test-slave-09fp"
server_key: "04997180282faba1a27f4a43ed27af27"
server_key_iv: "af6c68fbbf9d7053616c2b1d6d2684fc"
server_key_version: "encryptionkey@0"
I20260619 18:40:45.002383 5133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:45.029067 5133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:45.034400 5133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:45.034771 5133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:45.059791 5133 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39101
I20260619 18:40:45.059854 5209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39101 every 8 connection(s)
I20260619 18:40:45.061304 5133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:45.065863 5210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:45.071291 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5133
I20260619 18:40:45.071599 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:45.072268 3414 external_mini_cluster.cc:1428] Setting key 2eb35baa0205818b88556069c70d850d
I20260619 18:40:45.080076 5210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Bootstrap starting.
I20260619 18:40:45.084172 5210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:45.085489 5210 log.cc:824] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:45.089054 5210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: No bootstrap required, opened a new log
Jun 19 18:40:45 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:45.100217 5210 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } }
I20260619 18:40:45.100683 5210 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:45.100925 5210 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c231c9a7c544c8935cf3f23796be91, State: Initialized, Role: FOLLOWER
I20260619 18:40:45.101650 5210 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } }
I20260619 18:40:45.101819 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:45.074917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.3.85.190:39101)
0619 18:40:45.075863 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:45.075880 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:45.075945 (+ 65us) server_negotiation.cc:407] Connection header received
0619 18:40:45.078572 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:45.078608 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:45.079183 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:45.079703 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:45.081202 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.083348 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.084990 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.086218 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.086439 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:45.091556 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:45.091573 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:45.091603 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:45.091663 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:45.096123 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.097124 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.097137 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.097147 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.097425 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.097872 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.097878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.097882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.098859 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:45.099215 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:45.099722 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 18:40:45.100202 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":387,"threads_started":1}
I20260619 18:40:45.101958 5210 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:45.102123 5210 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:45.102370 5210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:45.105198 5210 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } }
I20260619 18:40:45.105882 5210 leader_election.cc:302] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4c231c9a7c544c8935cf3f23796be91; no voters:
I20260619 18:40:45.106942 5210 leader_election.cc:288] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:45.107333 5215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:45.108840 5215 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [term 1 LEADER]: Becoming Leader. State: Replica: e4c231c9a7c544c8935cf3f23796be91, State: Running, Role: LEADER
I20260619 18:40:45.109419 5215 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } }
I20260619 18:40:45.110013 5210 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:45.112470 5216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4c231c9a7c544c8935cf3f23796be91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } } }
I20260619 18:40:45.113147 5216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:45.116317 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c231c9a7c544c8935cf3f23796be91. Latest consensus state: current_term: 1 leader_uuid: "e4c231c9a7c544c8935cf3f23796be91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c231c9a7c544c8935cf3f23796be91" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39101 } } }
I20260619 18:40:45.116801 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:45.117249 5224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:45.122537 5224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:45.133011 5224 catalog_manager.cc:1347] Generated new cluster ID: 8ae781442b8a4ba0a6e2393d2c04c54a
I20260619 18:40:45.133169 5224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:45.165164 5224 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:45.166723 5224 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:45.175459 5224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Generated new TSK 0
I20260619 18:40:45.176463 5224 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:40:45.229789 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39101
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:45.586894 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:45.587299 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:45.587483 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:45.597389 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:45.597596 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:45.597756 5236 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:40:45.597864 5236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:45.597978 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:40:45.609175 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:45.610720 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:45.612216 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:45.627447 5241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:45.627525 5242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:45.627492 5244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:45.628688 5236 server_base.cc:1034] running on GCE node
I20260619 18:40:45.629364 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:45.630335 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:45.631606 5236 hybrid_clock.cc:648] HybridClock initialized: now 1781894445631546 us; error 53 us; skew 500 ppm
I20260619 18:40:45.632054 5236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:45 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:45.635492 5236 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:40:45.637290 5236 webserver.cc:460] Webserver started at http://127.3.85.129:44327/ using document root <none> and password file <none>
I20260619 18:40:45.638072 5236 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:45.638239 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:45.638667 5236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:45.641847 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "4a70499823a3405baaa10407fb9636ae"
format_stamp: "Formatted at 2026-06-19 18:40:45 on dist-test-slave-09fp"
server_key: "957976050079a760cbce96b0fca34e39"
server_key_iv: "fdc9a59e265974c8e5f0707c60d31719"
server_key_version: "encryptionkey@0"
I20260619 18:40:45.642856 5236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "4a70499823a3405baaa10407fb9636ae"
format_stamp: "Formatted at 2026-06-19 18:40:45 on dist-test-slave-09fp"
server_key: "957976050079a760cbce96b0fca34e39"
server_key_iv: "fdc9a59e265974c8e5f0707c60d31719"
server_key_version: "encryptionkey@0"
I20260619 18:40:45.648499 5236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:40:45.652587 5251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:45.654078 5236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:45.654297 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "4a70499823a3405baaa10407fb9636ae"
format_stamp: "Formatted at 2026-06-19 18:40:45 on dist-test-slave-09fp"
server_key: "957976050079a760cbce96b0fca34e39"
server_key_iv: "fdc9a59e265974c8e5f0707c60d31719"
server_key_version: "encryptionkey@0"
I20260619 18:40:45.654559 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:45.687292 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:45.692293 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:45.692660 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:45.695917 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:45.696043 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:45.696283 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:45.696393 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:45 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:45.725694 5261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:45.697143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.129:47399)
0619 18:40:45.698117 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:45.698182 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:45.700355 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:45.701564 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:45.701577 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:45.702511 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:45.703632 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:45.703664 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.705169 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.705174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:45.708134 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:45.708167 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.709603 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.709613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:45.710049 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:45.711087 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:45.711115 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:45.715041 (+ 3926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:45.719607 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:45.719623 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:45.719644 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:45.720483 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:45.721138 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:45.721142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:45.721145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:45.721381 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:45.722252 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:45.722259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:45.722819 (+ 560us) client_negotiation.cc:769] Sending connection context
0619 18:40:45.723272 (+ 453us) client_negotiation.cc:240] Negotiation successful
0619 18:40:45.724066 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":401,"threads_started":1}
I20260619 18:40:45.727377 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:45.696383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:47399 (local address 127.3.85.190:39101)
0619 18:40:45.696891 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:45.696898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:45.698362 (+ 1464us) server_negotiation.cc:407] Connection header received
0619 18:40:45.700815 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:45.700823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:45.700940 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:45.701128 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:45.703897 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.704955 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.708643 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.709298 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.709458 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:45.715365 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:45.715371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:45.715374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:45.715412 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:45.719397 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.720684 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.720696 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.720703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.720940 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.721533 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.721542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.721548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.722065 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:45.722304 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:45.726738 (+ 4434us) server_negotiation.cc:299] Negotiation successful
0619 18:40:45.727070 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
I20260619 18:40:45.750340 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:45.733584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:38557 (local address 127.3.85.190:39101)
0619 18:40:45.733752 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:45.733758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:45.734714 (+ 956us) server_negotiation.cc:407] Connection header received
0619 18:40:45.734955 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:45.734959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:45.735023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:45.735130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:45.736105 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.736825 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.738537 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.739173 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.739357 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:45.742214 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:45.742225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:45.742232 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:45.742283 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:45.746704 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.747805 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.747814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.747820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.748010 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:45.748649 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:45.748655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:45.748657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:45.748972 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:45.749116 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:45.749868 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 18:40:45.750068 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:40:45.751221 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:45.733654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.129:38557)
0619 18:40:45.734602 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:45.734619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:45.734812 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:45.735156 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:45.735161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:45.735650 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:45.735949 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:45.735958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.736969 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.736974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:45.738358 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:45.738373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:45.739494 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:45.739509 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:45.739852 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:45.740800 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:45.740833 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:45.741949 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:45.746939 (+ 4990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:45.746951 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:45.746957 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:45.747639 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:45.748176 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:45.748187 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:45.748193 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:45.748436 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:45.749121 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:45.749131 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:45.749448 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 18:40:45.749757 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 18:40:45.750298 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":538,"threads_started":1}
I20260619 18:40:45.754099 5236 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:46665
I20260619 18:40:45.754154 5371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:46665 every 8 connection(s)
I20260619 18:40:45.755743 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:45.758013 5372 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39101
I20260619 18:40:45.758337 5372 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:45.759210 5372 heartbeater.cc:511] Master 127.3.85.190:39101 requested a full tablet report, sending...
I20260619 18:40:45.761651 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5236
I20260619 18:40:45.761866 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:45.761817 5165 ts_manager.cc:194] Registered new tserver with Master: 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665)
I20260619 18:40:45.762723 3414 external_mini_cluster.cc:1428] Setting key bf535c2f2a538d4ae1e4bc9ad6896413
I20260619 18:40:45.763809 5165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:47399
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:40:45.806569 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39101
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:46.155135 5376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:46.155458 5376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:46.155633 5376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:46.163591 5376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:46.163733 5376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:46.163892 5376 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:40:46.163976 5376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:46.164115 5376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:40:46.174608 5376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:46.175995 5376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:46.177500 5376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:46.192286 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:46.192312 5382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:46.192257 5384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:46.193177 5376 server_base.cc:1034] running on GCE node
I20260619 18:40:46.193821 5376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:46.194897 5376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:46.196141 5376 hybrid_clock.cc:648] HybridClock initialized: now 1781894446196092 us; error 31 us; skew 500 ppm
I20260619 18:40:46.196455 5376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:46.200034 5376 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:40:46.201779 5376 webserver.cc:460] Webserver started at http://127.3.85.130:35305/ using document root <none> and password file <none>
I20260619 18:40:46.202610 5376 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:46.202759 5376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:46.203112 5376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:46.206231 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "cbfe69b9bad447e7955cb8e6ced37746"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "bdb17f5828546d79e7808a7c3b572e87"
server_key_iv: "c7d146091bd3b7ee87f55f01e20d064d"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.207242 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "cbfe69b9bad447e7955cb8e6ced37746"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "bdb17f5828546d79e7808a7c3b572e87"
server_key_iv: "c7d146091bd3b7ee87f55f01e20d064d"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.213279 5376 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:40:46.217464 5391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:46.219128 5376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:46.219329 5376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "cbfe69b9bad447e7955cb8e6ced37746"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "bdb17f5828546d79e7808a7c3b572e87"
server_key_iv: "c7d146091bd3b7ee87f55f01e20d064d"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.219527 5376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:46.267076 5376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:46.271749 5376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:46.272081 5376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:46.274971 5376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:46.275139 5376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:46.275300 5376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:46.275399 5376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:46.305694 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.275842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:56563 (local address 127.3.85.190:39101)
0619 18:40:46.276415 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.276423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.277676 (+ 1253us) server_negotiation.cc:407] Connection header received
0619 18:40:46.280191 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.280196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.280333 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.280431 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:46.283359 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.284567 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.288230 (+ 3663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.289067 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.289273 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.296332 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:46.296343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:46.296351 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:46.296409 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:46.300099 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.301623 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.301630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.301634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.301844 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.302636 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.302645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.302650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.303094 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:46.303308 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:46.304842 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0619 18:40:46.305176 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":325,"threads_started":1}
I20260619 18:40:46.307917 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.276543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.130:56563)
0619 18:40:46.277456 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.277514 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.279756 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.281075 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.281094 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.282001 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:46.283135 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.283158 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.284844 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.284853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.288011 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.288043 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.289429 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.289443 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.289909 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.291252 (+ 1343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:46.291299 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:46.295913 (+ 4614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:46.300357 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.300375 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.300404 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.301356 (+ 952us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.302026 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.302035 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.302041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.302417 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.303291 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:46.303310 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:46.304193 (+ 883us) client_negotiation.cc:769] Sending connection context
0619 18:40:46.304673 (+ 480us) client_negotiation.cc:240] Negotiation successful
0619 18:40:46.305531 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":420,"threads_started":1}
I20260619 18:40:46.320474 5376 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:39593
I20260619 18:40:46.320585 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:39593 every 8 connection(s)
I20260619 18:40:46.322142 5376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:46.324873 5512 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39101
I20260619 18:40:46.325151 5512 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:46.326015 5512 heartbeater.cc:511] Master 127.3.85.190:39101 requested a full tablet report, sending...
I20260619 18:40:46.327683 5164 ts_manager.cc:194] Registered new tserver with Master: cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593)
I20260619 18:40:46.329170 5164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:56563
I20260619 18:40:46.331475 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5376
I20260619 18:40:46.331672 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:46.332238 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.317384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:50243 (local address 127.3.85.190:39101)
0619 18:40:46.317561 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.317568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.317726 (+ 158us) server_negotiation.cc:407] Connection header received
0619 18:40:46.317942 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.317947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.318066 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.318154 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:46.319800 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.320777 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.322301 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.322917 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.323065 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.325804 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:46.325809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:46.325813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:46.325849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:46.329071 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.330058 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.330063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.330067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.330224 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.330748 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.330751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.330754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.331063 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:46.331257 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:46.331657 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 18:40:46.331893 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:40:46.332247 3414 external_mini_cluster.cc:1428] Setting key 979b5572027e4753cdaaa056117d04ad
I20260619 18:40:46.332360 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.316910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.130:50243)
0619 18:40:46.317561 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.317587 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.317763 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.318415 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.318419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.319072 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:46.319623 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.319637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.320994 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.321005 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.322121 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.322139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.323360 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.323368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.323610 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.324474 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:46.324502 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:46.325580 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:46.329280 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.329287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.329291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.329919 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.330367 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.330373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.330376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.330612 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.331194 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:46.331203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:46.331407 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 18:40:46.331564 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:40:46.331900 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":667,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:40:46.374883 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39101
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:46.721688 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:46.721997 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:46.722178 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:46.730103 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:46.730237 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:46.730419 5516 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:40:46.730536 5516 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:40:46.730655 5516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:40:46.741148 5516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:46.742704 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:46.744107 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:46.758695 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:46.759539 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:46.758695 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:46.762610 5516 server_base.cc:1034] running on GCE node
I20260619 18:40:46.763365 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:46.764505 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:46.765733 5516 hybrid_clock.cc:648] HybridClock initialized: now 1781894446765675 us; error 52 us; skew 500 ppm
I20260619 18:40:46.766103 5516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:40:46.767433 5372 heartbeater.cc:503] Master 127.3.85.190:39101 was elected leader, sending a full tablet report...
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:46.769906 5516 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:40:46.771710 5516 webserver.cc:460] Webserver started at http://127.3.85.131:40945/ using document root <none> and password file <none>
I20260619 18:40:46.772509 5516 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:46.772647 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:46.772990 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:46.776468 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "a0226b6ff32241a89481d01bbbc2d5ac"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "af22ba74cc11e214ea4fc03c12e55443"
server_key_iv: "c4f25625ac9bb8709a8fe9a15b1d5717"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.777326 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "a0226b6ff32241a89481d01bbbc2d5ac"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "af22ba74cc11e214ea4fc03c12e55443"
server_key_iv: "c4f25625ac9bb8709a8fe9a15b1d5717"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.784662 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 18:40:46.789124 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:46.790958 5516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:46.791193 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "a0226b6ff32241a89481d01bbbc2d5ac"
format_stamp: "Formatted at 2026-06-19 18:40:46 on dist-test-slave-09fp"
server_key: "af22ba74cc11e214ea4fc03c12e55443"
server_key_iv: "c4f25625ac9bb8709a8fe9a15b1d5717"
server_key_version: "encryptionkey@0"
I20260619 18:40:46.791436 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:46.817427 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:46.824361 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:46.824747 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:46.828037 5516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:46.828214 5516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:46.828354 5516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:46.828449 5516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:46.856981 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.828858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:38421 (local address 127.3.85.190:39101)
0619 18:40:46.829087 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.829093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.831821 (+ 2728us) server_negotiation.cc:407] Connection header received
0619 18:40:46.835699 (+ 3878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.835708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.835819 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.835945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:46.839096 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.840404 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.843246 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.844094 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.844292 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.849383 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:46.849394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:46.849399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:46.849439 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:46.853047 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.854317 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.854325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.854329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.854554 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.855071 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.855076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.855079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.855406 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:46.855576 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:46.856368 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 18:40:46.856605 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 18:40:46.858085 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.830060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.131:38421)
0619 18:40:46.831538 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.831615 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.835190 (+ 3575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.836327 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.836345 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.837723 (+ 1378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:46.838743 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.838800 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.840615 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.840619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.842990 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.843020 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.844430 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.844445 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.844753 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.845667 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:46.845699 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:46.849013 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:46.853235 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.853247 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.853272 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.854050 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.854685 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.854690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.854700 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.854927 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.855552 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:46.855562 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:46.855975 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 18:40:46.856263 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 18:40:46.856831 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":697,"threads_started":1}
I20260619 18:40:46.884447 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.864896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.3.85.131:58831)
0619 18:40:46.865725 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.865755 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.865964 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.866376 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.866384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.867277 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:46.867837 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.867858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.869039 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.869048 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.871495 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.871526 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.873300 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.873310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:46.873528 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.874544 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:46.874586 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:46.876467 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:46.880734 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.880740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.880746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.881436 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.882039 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:46.882050 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:46.882058 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:46.882312 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:46.883251 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:46.883257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:46.883529 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 18:40:46.883725 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:40:46.884024 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":402,"threads_started":1}
I20260619 18:40:46.884685 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.864801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:58831 (local address 127.3.85.190:39101)
0619 18:40:46.865037 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.865046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.865894 (+ 848us) server_negotiation.cc:407] Connection header received
0619 18:40:46.866110 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.866114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.866191 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.866302 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:46.868090 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.868852 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.871792 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.872796 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.872938 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.876826 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:46.876834 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:46.876839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:46.876890 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:46.880472 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.881595 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.881601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.881605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.881846 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.882669 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.882677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.882683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.883113 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:46.883286 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:46.883880 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:40:46.884146 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:40:46.886134 5516 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:42317
I20260619 18:40:46.886262 5650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:42317 every 8 connection(s)
I20260619 18:40:46.887737 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:46.890745 5651 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39101
I20260619 18:40:46.891417 5651 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:46.892100 5651 heartbeater.cc:511] Master 127.3.85.190:39101 requested a full tablet report, sending...
I20260619 18:40:46.893805 5165 ts_manager.cc:194] Registered new tserver with Master: a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317)
I20260619 18:40:46.894842 5165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:38421
I20260619 18:40:46.897135 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5516
I20260619 18:40:46.897327 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:46.897773 3414 external_mini_cluster.cc:1428] Setting key 8508905ee63bc83ec065ea1638cf7e69
I20260619 18:40:46.901306 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:40:46 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:46.954718 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.938346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.3.85.190:39101)
0619 18:40:46.938639 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.938646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.938945 (+ 299us) server_negotiation.cc:407] Connection header received
0619 18:40:46.939180 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.939185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.939251 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.939362 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:46.940841 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.941594 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.943008 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.943424 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.943586 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:46.948091 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:46.948096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:46.948100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:46.948146 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:46.951092 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.952227 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.952231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.952233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.952390 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:46.953179 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:46.953189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:46.953192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:46.953502 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:46.953747 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:46.954099 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 18:40:46.954308 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:40:46.960158 5165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:46.967746 5165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:47.005264 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.981680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:41470)
0619 18:40:46.993433 (+ 11753us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.993513 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.993809 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.997048 (+ 3239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.997059 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.997100 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:46.997597 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.997617 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.999750 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.999756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.002561 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.002577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.004064 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.004274 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.004332 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.004522 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.004752 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11370,"mutex_wait_us":114,"thread_start_us":12592,"threads_started":1}
I20260619 18:40:47.005340 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.982011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42317 (local address 127.0.0.1:48076)
0619 18:40:46.993933 (+ 11922us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.993967 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.994147 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.997862 (+ 3715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.997867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.997899 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:46.998363 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.998376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.000430 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.000440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.002631 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.002648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.004582 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.004757 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.004786 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.004871 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.005018 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11757,"thread_start_us":12735,"threads_started":1}
I20260619 18:40:47.006064 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.982281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.0.0.1:39424)
0619 18:40:46.995153 (+ 12872us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:46.995189 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:46.995367 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:46.997389 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:46.997395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:46.997421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:46.997846 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:46.997857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.999591 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:46.999596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.002969 (+ 3373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.002980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.004936 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.005107 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.005129 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.005253 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.005698 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12658,"mutex_wait_us":196,"thread_start_us":10607,"threads_started":1}
I20260619 18:40:47.006217 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.981816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.3.85.130:39593)
0619 18:40:46.996141 (+ 14325us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.996156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.996243 (+ 87us) server_negotiation.cc:407] Connection header received
0619 18:40:46.996363 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.996371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.996723 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.997001 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:46.997881 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.999606 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.002799 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.003846 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004032 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.004271 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.005732 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.005904 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14013,"thread_start_us":13848,"threads_started":1}
I20260619 18:40:47.006259 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.982765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.3.85.131:42317)
0619 18:40:46.996929 (+ 14164us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.996938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.996970 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:40:46.997062 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.997069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.997600 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.998596 (+ 996us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:46.998950 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.000280 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.002845 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.004362 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004517 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.004752 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.005748 (+ 996us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.005906 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13976,"thread_start_us":13807,"threads_started":1}
I20260619 18:40:47.006424 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:46.982945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.3.85.129:46665)
0619 18:40:46.996540 (+ 13595us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:46.996553 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:46.996590 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:40:46.996685 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:46.996692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:46.996948 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:46.997168 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:46.998056 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:46.999433 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.003132 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.004608 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.004749 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.004950 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.005300 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.005669 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13380,"thread_start_us":13255,"threads_started":1}
I20260619 18:40:47.010093 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 718efb5ce3334452b3547d68b3a59d0f (DEFAULT_TABLE table=test-table [id=28409c8b442f4bb09835685f1b7bc064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.010200 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 718efb5ce3334452b3547d68b3a59d0f (DEFAULT_TABLE table=test-table [id=28409c8b442f4bb09835685f1b7bc064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.010210 5581 tablet_service.cc:1467] Processing CreateTablet for tablet 718efb5ce3334452b3547d68b3a59d0f (DEFAULT_TABLE table=test-table [id=28409c8b442f4bb09835685f1b7bc064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.011754 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718efb5ce3334452b3547d68b3a59d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.011812 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718efb5ce3334452b3547d68b3a59d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.011823 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718efb5ce3334452b3547d68b3a59d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.024717 5670 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Bootstrap starting.
I20260619 18:40:47.027140 5671 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Bootstrap starting.
I20260619 18:40:47.028877 5670 tablet_bootstrap.cc:654] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.028980 5672 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Bootstrap starting.
I20260619 18:40:47.030573 5670 log.cc:824] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:47.031226 5671 tablet_bootstrap.cc:654] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.032480 5672 tablet_bootstrap.cc:654] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.032915 5671 log.cc:824] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:47.033581 5672 log.cc:824] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:47.036680 5672 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: No bootstrap required, opened a new log
I20260619 18:40:47.036736 5671 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: No bootstrap required, opened a new log
I20260619 18:40:47.036734 5670 tablet_bootstrap.cc:492] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: No bootstrap required, opened a new log
I20260619 18:40:47.036996 5672 ts_tablet_manager.cc:1397] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:40:47.037011 5671 ts_tablet_manager.cc:1397] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 18:40:47.037011 5670 ts_tablet_manager.cc:1397] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 18:40:47.045195 5672 raft_consensus.cc:348] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.045644 5672 raft_consensus.cc:374] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.045742 5672 raft_consensus.cc:729] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.046512 5672 consensus_queue.cc:260] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.047719 5651 heartbeater.cc:503] Master 127.3.85.190:39101 was elected leader, sending a full tablet report...
I20260619 18:40:47.048470 5672 ts_tablet_manager.cc:1428] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 18:40:47.048583 5670 raft_consensus.cc:348] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.048497 5671 raft_consensus.cc:348] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.048993 5671 raft_consensus.cc:374] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.048993 5670 raft_consensus.cc:374] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.049094 5671 raft_consensus.cc:729] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a70499823a3405baaa10407fb9636ae, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.049098 5670 raft_consensus.cc:729] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.049777 5670 consensus_queue.cc:260] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.049786 5671 consensus_queue.cc:260] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.052265 5671 ts_tablet_manager.cc:1428] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260619 18:40:47.052356 5670 ts_tablet_manager.cc:1428] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260619 18:40:47.052364 5512 heartbeater.cc:503] Master 127.3.85.190:39101 was elected leader, sending a full tablet report...
W20260619 18:40:47.074856 5513 tablet.cc:2369] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:47.139817 5652 tablet.cc:2369] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:47.217338 5676 raft_consensus.cc:482] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:47.217676 5676 raft_consensus.cc:504] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.219484 5676 leader_election.cc:288] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593), 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665)
I20260619 18:40:47.225687 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.219933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:57597 (local address 127.3.85.129:46665)
0619 18:40:47.220157 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.220163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.220187 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:40:47.220308 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.220316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.220420 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.220539 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:47.220974 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.221695 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.223537 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.224706 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.224927 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.225008 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.225295 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.225419 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 18:40:47.225716 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.219838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.3.85.131:57597)
0619 18:40:47.220019 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.220041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.220173 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.220580 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.220586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.220624 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:47.220814 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.220822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.221829 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.221838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.223364 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.223376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.224901 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.224908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.225122 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.225161 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.225267 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.225419 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":34}
I20260619 18:40:47.226342 5665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.220049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:35525 (local address 127.3.85.130:39593)
0619 18:40:47.220225 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.220231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.220849 (+ 618us) server_negotiation.cc:407] Connection header received
0619 18:40:47.221050 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.221055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.221111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.221204 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:47.221736 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.222503 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.224334 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.225326 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.225474 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.225540 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.225958 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.226109 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:40:47.227118 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.219838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.3.85.131:35525)
0619 18:40:47.220735 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.220750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.220902 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.221229 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.221233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.221257 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:47.221574 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.221583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.222644 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.222650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.224143 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.224154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.225546 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.225553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.225750 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.225786 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.226558 (+ 772us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.226718 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":212,"thread_start_us":319,"threads_started":1}
I20260619 18:40:47.226912 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718efb5ce3334452b3547d68b3a59d0f" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae" is_pre_election: true
I20260619 18:40:47.227546 5322 raft_consensus.cc:2454] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:47.228240 5533 leader_election.cc:302] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, a0226b6ff32241a89481d01bbbc2d5ac; no voters:
I20260619 18:40:47.228295 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718efb5ce3334452b3547d68b3a59d0f" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746" is_pre_election: true
I20260619 18:40:47.228643 5676 raft_consensus.cc:2781] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:47.228798 5676 raft_consensus.cc:482] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:47.228892 5676 raft_consensus.cc:3037] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.228920 5462 raft_consensus.cc:2454] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:47.231525 5676 raft_consensus.cc:504] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.232208 5676 leader_election.cc:288] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Requested vote from peers cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593), 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665)
I20260619 18:40:47.232723 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718efb5ce3334452b3547d68b3a59d0f" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746"
I20260619 18:40:47.232769 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718efb5ce3334452b3547d68b3a59d0f" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae"
I20260619 18:40:47.232944 5462 raft_consensus.cc:3037] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.233003 5322 raft_consensus.cc:3037] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.235423 5322 raft_consensus.cc:2454] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:47.235558 5462 raft_consensus.cc:2454] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:47.236047 5532 leader_election.cc:302] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0226b6ff32241a89481d01bbbc2d5ac, cbfe69b9bad447e7955cb8e6ced37746; no voters:
I20260619 18:40:47.236454 5676 raft_consensus.cc:2781] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:47.236981 5676 raft_consensus.cc:686] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Becoming Leader. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Running, Role: LEADER
I20260619 18:40:47.237550 5676 consensus_queue.cc:237] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } }
I20260619 18:40:47.243227 5163 catalog_manager.cc:5495] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131). New cstate: current_term: 1 leader_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } health_report { overall_health: UNKNOWN } } }
W20260619 18:40:47.265021 5373 tablet.cc:2369] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:40:47 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894444, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:47.359412 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.345175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39101 (local address 127.0.0.1:60826)
0619 18:40:47.345802 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.345818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.346016 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.346376 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.346380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.347174 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:47.347437 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.347448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.348472 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.348480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.349456 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.349468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.350210 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.350216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.350378 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.351142 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:47.351161 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:47.353553 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:47.356671 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:47.356684 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:47.356691 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:47.357198 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:47.357581 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:47.357584 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:47.357586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:47.357735 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:47.358268 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:47.358282 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:47.358576 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.358776 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.359021 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":341,"threads_started":1}
I20260619 18:40:47.359436 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.345124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.3.85.190:39101)
0619 18:40:47.345312 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.345316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.345920 (+ 604us) server_negotiation.cc:407] Connection header received
0619 18:40:47.346139 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.346145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.346244 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.346353 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:47.347622 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.348317 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.349610 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.350013 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.350159 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.353793 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:47.353798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:47.353801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:47.353832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:47.356484 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:47.357325 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:47.357329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:47.357331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:47.357455 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:47.357828 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:47.357831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:47.357833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:47.358108 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:47.358239 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.358906 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.359165 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":73}
I20260619 18:40:47.365257 5163 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:60826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:40:47.366556 5163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:47.374049 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 10e3553c3d0540af8a3248847258d921 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a132eea2ea84c9cbf831d3378eaedcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:47.374022 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 10e3553c3d0540af8a3248847258d921 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a132eea2ea84c9cbf831d3378eaedcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:47.374354 5581 tablet_service.cc:1467] Processing CreateTablet for tablet 10e3553c3d0540af8a3248847258d921 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a132eea2ea84c9cbf831d3378eaedcb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:40:47.375125 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e3553c3d0540af8a3248847258d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.375151 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e3553c3d0540af8a3248847258d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.375344 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e3553c3d0540af8a3248847258d921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.380755 5671 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: Bootstrap starting.
I20260619 18:40:47.382162 5671 tablet_bootstrap.cc:654] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.382704 5670 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: Bootstrap starting.
I20260619 18:40:47.382719 5672 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: Bootstrap starting.
I20260619 18:40:47.384534 5670 tablet_bootstrap.cc:654] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.384827 5672 tablet_bootstrap.cc:654] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.385241 5671 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: No bootstrap required, opened a new log
I20260619 18:40:47.385421 5671 ts_tablet_manager.cc:1397] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:40:47.386534 5671 raft_consensus.cc:348] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.386823 5671 raft_consensus.cc:374] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.386924 5671 raft_consensus.cc:729] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a70499823a3405baaa10407fb9636ae, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.387279 5672 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: No bootstrap required, opened a new log
I20260619 18:40:47.387188 5671 consensus_queue.cc:260] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.387437 5672 ts_tablet_manager.cc:1397] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:40:47.387702 5677 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.387961 5670 tablet_bootstrap.cc:492] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: No bootstrap required, opened a new log
I20260619 18:40:47.388079 5677 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.388324 5672 raft_consensus.cc:348] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.388453 5670 ts_tablet_manager.cc:1397] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 18:40:47.388567 5672 raft_consensus.cc:374] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.388669 5672 raft_consensus.cc:729] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.388908 5672 consensus_queue.cc:260] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.389339 5671 ts_tablet_manager.cc:1428] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 18:40:47.389484 5672 ts_tablet_manager.cc:1428] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:47.389726 5676 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.389995 5676 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.389997 5670 raft_consensus.cc:348] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.390280 5670 raft_consensus.cc:374] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:47.390396 5670 raft_consensus.cc:729] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Initialized, Role: FOLLOWER
I20260619 18:40:47.390699 5670 consensus_queue.cc:260] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.391219 5678 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.391618 5678 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.392416 5670 ts_tablet_manager.cc:1428] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:40:47.629892 5676 consensus_queue.cc:1035] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:47.638630 5680 consensus_queue.cc:1035] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:47.640710 5678 raft_consensus.cc:482] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:47.640996 5678 raft_consensus.cc:504] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.643322 5678 leader_election.cc:288] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317)
I20260619 18:40:47.660282 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.644092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:42083 (local address 127.3.85.129:46665)
0619 18:40:47.644352 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.644361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.645769 (+ 1408us) server_negotiation.cc:407] Connection header received
0619 18:40:47.646128 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.646136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.646257 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.646334 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:47.648873 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.650322 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.655512 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.657554 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.657893 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.658044 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.659500 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.659783 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:40:47.661927 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.643549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.3.85.130:42083)
0619 18:40:47.645626 (+ 2077us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.645652 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.645911 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.646626 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.646634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.646672 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:47.647168 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.647183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.650609 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.650619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.653509 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.653539 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.658642 (+ 5103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.658654 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.659109 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.659177 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.660986 (+ 1809us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.661240 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"thread_start_us":16639,"threads_started":1}
I20260619 18:40:47.662945 5321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e3553c3d0540af8a3248847258d921" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae" is_pre_election: true
I20260619 18:40:47.663358 5321 raft_consensus.cc:2454] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 0.
I20260619 18:40:47.664005 5393 leader_election.cc:302] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, cbfe69b9bad447e7955cb8e6ced37746; no voters:
I20260619 18:40:47.664685 5695 raft_consensus.cc:2781] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:47.664834 5695 raft_consensus.cc:482] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:47.664988 5695 raft_consensus.cc:3037] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.668435 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.643604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:46611 (local address 127.3.85.131:42317)
0619 18:40:47.660159 (+ 16555us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.660168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.660571 (+ 403us) server_negotiation.cc:407] Connection header received
0619 18:40:47.660880 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.660888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.660977 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.661084 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:47.662234 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.663356 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.665781 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.667080 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.667298 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.667376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.667886 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.668033 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16397,"thread_start_us":16257,"threads_started":1}
I20260619 18:40:47.668356 5695 raft_consensus.cc:504] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.669166 5695 leader_election.cc:288] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 election: Requested vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317)
I20260619 18:40:47.669564 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.644677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42317 (local address 127.3.85.130:46611)
0619 18:40:47.660442 (+ 15765us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.660464 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.660646 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.661523 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.661529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.661557 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:47.662011 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.662029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.663531 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.663536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.665602 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.665619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.667567 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.667573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.667716 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.667741 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.667859 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.669097 (+ 1238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15630,"thread_start_us":433,"threads_started":1}
I20260619 18:40:47.669669 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e3553c3d0540af8a3248847258d921" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae"
I20260619 18:40:47.669934 5322 raft_consensus.cc:3037] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.670661 5605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e3553c3d0540af8a3248847258d921" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" is_pre_election: true
I20260619 18:40:47.670753 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e3553c3d0540af8a3248847258d921" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0226b6ff32241a89481d01bbbc2d5ac"
I20260619 18:40:47.671046 5605 raft_consensus.cc:2454] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 0.
I20260619 18:40:47.672412 5322 raft_consensus.cc:2454] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 1.
I20260619 18:40:47.672878 5393 leader_election.cc:302] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, cbfe69b9bad447e7955cb8e6ced37746; no voters: a0226b6ff32241a89481d01bbbc2d5ac
I20260619 18:40:47.673224 5695 raft_consensus.cc:2781] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:47.673440 5695 raft_consensus.cc:686] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 LEADER]: Becoming Leader. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Running, Role: LEADER
I20260619 18:40:47.673871 5695 consensus_queue.cc:237] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:47.675922 5678 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: TxnStatusTablet state changed. Reason: New leader cbfe69b9bad447e7955cb8e6ced37746. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.676306 5678 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:47.677011 5702 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:40:47.677328 5702 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:40:47.678059 5163 catalog_manager.cc:5495] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:47.771380 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.762774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.3.85.130:39593)
0619 18:40:47.763393 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.763399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.763603 (+ 204us) server_negotiation.cc:407] Connection header received
0619 18:40:47.763939 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.763945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.764056 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.764385 (+ 329us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:47.764863 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.766066 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.768202 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.768901 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.769057 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.769712 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.770570 (+ 858us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.770724 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.770875 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.770970 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":327,"threads_started":1}
I20260619 18:40:47.771503 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.763260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:41474)
0619 18:40:47.763487 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.763509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.763705 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.764431 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.764437 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.764453 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:47.764684 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.764696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.766265 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.766272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.767987 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.768010 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.769126 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.769133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.769375 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.769466 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.770709 (+ 1243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.770731 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.770831 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.771051 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:40:47.779330 5322 raft_consensus.cc:1261] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Refusing update from remote peer cbfe69b9bad447e7955cb8e6ced37746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:47.779449 5604 raft_consensus.cc:3037] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:47.780123 5695 consensus_queue.cc:1035] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:47.782958 5604 raft_consensus.cc:1261] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Refusing update from remote peer cbfe69b9bad447e7955cb8e6ced37746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:47.783865 5695 consensus_queue.cc:1035] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:47.792084 5693 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: TxnStatusTablet state changed. Reason: New leader cbfe69b9bad447e7955cb8e6ced37746. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.792513 5693 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.792615 5677 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: TxnStatusTablet state changed. Reason: New leader cbfe69b9bad447e7955cb8e6ced37746. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.792968 5677 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.795858 5700 mvcc.cc:205] Tried to move back new op lower bound from 7298639658095382528 to 7298639657676210176. Current Snapshot: MvccSnapshot[applied={T|T < 7298639658095382528}]
I20260619 18:40:47.799468 5692 mvcc.cc:205] Tried to move back new op lower bound from 7298639658095382528 to 7298639657676210176. Current Snapshot: MvccSnapshot[applied={T|T < 7298639658095382528}]
I20260619 18:40:47.800570 5695 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.800922 5695 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:47.801196 5695 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.801393 5695 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P cbfe69b9bad447e7955cb8e6ced37746: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:40:47.814394 5693 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.814718 5693 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P a0226b6ff32241a89481d01bbbc2d5ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.824010 5677 tablet_replica.cc:439] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } } }
I20260619 18:40:47.824329 5677 tablet_replica.cc:442] T 10e3553c3d0540af8a3248847258d921 P 4a70499823a3405baaa10407fb9636ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:40:47.833334 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.818742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.3.85.131:42317)
0619 18:40:47.819000 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.819009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.827050 (+ 8041us) server_negotiation.cc:407] Connection header received
0619 18:40:47.827304 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.827311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.827428 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.827603 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:47.827999 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.829168 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.830817 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.831447 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.831591 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.831969 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.832735 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.832880 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.832937 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.833026 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:40:47.852102 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.845689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.3.85.131:51233)
0619 18:40:47.846370 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.846386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.846587 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.846970 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.846973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.846982 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:47.847231 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.847239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.848769 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.848773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.849855 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.849868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.850925 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.850930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.851058 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.851121 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.851594 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.851604 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.851683 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.851814 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":426,"threads_started":1}
I20260619 18:40:47.852139 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.845815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:51233 (local address 127.3.85.130:39593)
0619 18:40:47.845985 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.845990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.846501 (+ 511us) server_negotiation.cc:407] Connection header received
0619 18:40:47.846742 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.846748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.846845 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.846949 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:47.847377 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.848603 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.850025 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.850736 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.850879 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.851330 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.851479 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.851593 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.851730 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.851823 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:40:47.897310 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.887493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:43953 (local address 127.3.85.131:42317)
0619 18:40:47.890651 (+ 3158us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:47.890655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:47.890674 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:40:47.890789 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:47.890794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:47.890889 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:47.891008 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:47.891993 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.892898 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.895092 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.895591 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.895786 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.896379 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.896549 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.896630 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:47.896923 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 18:40:47.897012 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3039}
I20260619 18:40:47.898583 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:47.887489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42317 (local address 127.3.85.130:43953)
0619 18:40:47.888085 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:47.888117 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:47.888265 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:47.891364 (+ 3099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:47.891370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:47.891385 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:47.891752 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.891771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.893492 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.893500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.894944 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:47.894958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:47.895808 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:47.895815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:47.896042 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:47.896134 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:47.896767 (+ 633us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:47.896780 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:40:47.897999 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0619 18:40:47.898169 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":2177,"threads_started":1}
I20260619 18:40:47.947266 5163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:47.947594 5163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:47.952210 5163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Sending DeleteTablet for 3 replicas of tablet 718efb5ce3334452b3547d68b3a59d0f
I20260619 18:40:47.953675 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 718efb5ce3334452b3547d68b3a59d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:47 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:41470
I20260619 18:40:47.953680 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 718efb5ce3334452b3547d68b3a59d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:47 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39424
I20260619 18:40:47.954071 5581 tablet_service.cc:1514] Processing DeleteTablet for tablet 718efb5ce3334452b3547d68b3a59d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:47 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:48076
I20260619 18:40:47.957130 5163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:47.958819 5163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:47.967051 5721 tablet_replica.cc:330] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: stopping tablet replica
I20260619 18:40:47.967635 5721 raft_consensus.cc:2229] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:47.968030 5721 raft_consensus.cc:2258] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:47.964655 5722 tablet_replica.cc:330] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: stopping tablet replica
I20260619 18:40:47.971642 5723 tablet_replica.cc:330] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: stopping tablet replica
I20260619 18:40:47.972790 5302 tablet_service.cc:1467] Processing CreateTablet for tablet c780dc1539cf4fcd95065339682c15fd (DEFAULT_TABLE table=test-table [id=abe8bbcf69d1442bad158f95c7a9980b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.972961 5580 tablet_service.cc:1467] Processing CreateTablet for tablet c780dc1539cf4fcd95065339682c15fd (DEFAULT_TABLE table=test-table [id=abe8bbcf69d1442bad158f95c7a9980b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.973631 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c780dc1539cf4fcd95065339682c15fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.971688 5442 tablet_service.cc:1467] Processing CreateTablet for tablet c780dc1539cf4fcd95065339682c15fd (DEFAULT_TABLE table=test-table [id=abe8bbcf69d1442bad158f95c7a9980b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:47.974185 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c780dc1539cf4fcd95065339682c15fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.974216 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c780dc1539cf4fcd95065339682c15fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:47.975857 5721 ts_tablet_manager.cc:1905] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:47.984762 5721 ts_tablet_manager.cc:1918] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:47.970405 5722 raft_consensus.cc:2229] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:47.985009 5721 log.cc:1196] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/718efb5ce3334452b3547d68b3a59d0f
I20260619 18:40:47.985327 5722 raft_consensus.cc:2258] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:47.985652 5721 ts_tablet_manager.cc:1939] T 718efb5ce3334452b3547d68b3a59d0f P 4a70499823a3405baaa10407fb9636ae: Deleting consensus metadata
I20260619 18:40:47.988796 5724 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Bootstrap starting.
I20260619 18:40:47.972118 5723 raft_consensus.cc:2229] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:47.990100 5723 raft_consensus.cc:2258] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:47.990927 5725 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Bootstrap starting.
I20260619 18:40:47.991144 5724 tablet_bootstrap.cc:654] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.992189 5722 ts_tablet_manager.cc:1905] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:47.992995 5725 tablet_bootstrap.cc:654] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:47.997869 5723 ts_tablet_manager.cc:1905] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:48.004717 5725 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: No bootstrap required, opened a new log
I20260619 18:40:48.004925 5725 ts_tablet_manager.cc:1397] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260619 18:40:48.005704 5728 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Bootstrap starting.
I20260619 18:40:48.005795 5725 raft_consensus.cc:348] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.006142 5725 raft_consensus.cc:374] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:48.006258 5725 raft_consensus.cc:729] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Initialized, Role: FOLLOWER
I20260619 18:40:48.006561 5725 consensus_queue.cc:260] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.007944 5725 ts_tablet_manager.cc:1428] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:48.009642 5150 catalog_manager.cc:4841] TS 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665): tablet 718efb5ce3334452b3547d68b3a59d0f (table test-table [id=28409c8b442f4bb09835685f1b7bc064]) successfully deleted
I20260619 18:40:48.012064 5728 tablet_bootstrap.cc:654] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:48.015431 5722 ts_tablet_manager.cc:1918] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:48.015671 5722 log.cc:1196] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/718efb5ce3334452b3547d68b3a59d0f
I20260619 18:40:48.016323 5722 ts_tablet_manager.cc:1939] T 718efb5ce3334452b3547d68b3a59d0f P cbfe69b9bad447e7955cb8e6ced37746: Deleting consensus metadata
I20260619 18:40:48.018493 5149 catalog_manager.cc:4841] TS cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593): tablet 718efb5ce3334452b3547d68b3a59d0f (table test-table [id=28409c8b442f4bb09835685f1b7bc064]) successfully deleted
I20260619 18:40:48.018958 5724 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: No bootstrap required, opened a new log
I20260619 18:40:48.019196 5724 ts_tablet_manager.cc:1397] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.011s
I20260619 18:40:48.020047 5724 raft_consensus.cc:348] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.020388 5724 raft_consensus.cc:374] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:48.020530 5724 raft_consensus.cc:729] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a70499823a3405baaa10407fb9636ae, State: Initialized, Role: FOLLOWER
I20260619 18:40:48.020916 5724 consensus_queue.cc:260] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.021807 5723 ts_tablet_manager.cc:1918] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:48.022011 5723 log.cc:1196] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/718efb5ce3334452b3547d68b3a59d0f
I20260619 18:40:48.022656 5723 ts_tablet_manager.cc:1939] T 718efb5ce3334452b3547d68b3a59d0f P a0226b6ff32241a89481d01bbbc2d5ac: Deleting consensus metadata
I20260619 18:40:48.024637 5724 ts_tablet_manager.cc:1428] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:40:48.025701 5152 catalog_manager.cc:4841] TS a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317): tablet 718efb5ce3334452b3547d68b3a59d0f (table test-table [id=28409c8b442f4bb09835685f1b7bc064]) successfully deleted
I20260619 18:40:48.026361 5728 tablet_bootstrap.cc:492] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: No bootstrap required, opened a new log
I20260619 18:40:48.026557 5728 ts_tablet_manager.cc:1397] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260619 18:40:48.027432 5728 raft_consensus.cc:348] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.027693 5728 raft_consensus.cc:374] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:48.027789 5728 raft_consensus.cc:729] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Initialized, Role: FOLLOWER
I20260619 18:40:48.028090 5728 consensus_queue.cc:260] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.028952 5728 ts_tablet_manager.cc:1428] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:48.306802 5678 raft_consensus.cc:482] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:48.307081 5678 raft_consensus.cc:504] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.307951 5678 leader_election.cc:288] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317)
I20260619 18:40:48.308548 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c780dc1539cf4fcd95065339682c15fd" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae" is_pre_election: true
I20260619 18:40:48.308640 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c780dc1539cf4fcd95065339682c15fd" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" is_pre_election: true
I20260619 18:40:48.308838 5322 raft_consensus.cc:2454] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 0.
I20260619 18:40:48.308924 5604 raft_consensus.cc:2454] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 0.
I20260619 18:40:48.309451 5395 leader_election.cc:302] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, a0226b6ff32241a89481d01bbbc2d5ac, cbfe69b9bad447e7955cb8e6ced37746; no voters:
I20260619 18:40:48.309947 5678 raft_consensus.cc:2781] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:48.310082 5678 raft_consensus.cc:482] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:48.310236 5678 raft_consensus.cc:3037] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:48.312422 5678 raft_consensus.cc:504] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.313058 5678 leader_election.cc:288] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 election: Requested vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317)
I20260619 18:40:48.313709 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c780dc1539cf4fcd95065339682c15fd" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0226b6ff32241a89481d01bbbc2d5ac"
I20260619 18:40:48.313705 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c780dc1539cf4fcd95065339682c15fd" candidate_uuid: "cbfe69b9bad447e7955cb8e6ced37746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae"
I20260619 18:40:48.313958 5322 raft_consensus.cc:3037] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:48.313966 5604 raft_consensus.cc:3037] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:48.316582 5604 raft_consensus.cc:2454] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 1.
I20260619 18:40:48.317101 5322 raft_consensus.cc:2454] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbfe69b9bad447e7955cb8e6ced37746 in term 1.
I20260619 18:40:48.317107 5395 leader_election.cc:302] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0226b6ff32241a89481d01bbbc2d5ac, cbfe69b9bad447e7955cb8e6ced37746; no voters:
I20260619 18:40:48.317473 5678 raft_consensus.cc:2781] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:48.317648 5678 raft_consensus.cc:686] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 1 LEADER]: Becoming Leader. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Running, Role: LEADER
I20260619 18:40:48.317934 5678 consensus_queue.cc:237] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:48.320950 5165 catalog_manager.cc:5495] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "cbfe69b9bad447e7955cb8e6ced37746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:48.367354 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:48.362145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.3.85.130:39593)
0619 18:40:48.362771 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:48.362779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:48.362800 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:40:48.362891 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:48.362896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:48.362967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:48.363076 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:48.363482 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.364214 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.365533 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.365997 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.366148 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:48.366583 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:48.366698 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:48.366799 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:48.366945 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:40:48.367023 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":402,"threads_started":1}
I20260619 18:40:48.382515 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:48.377206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:39253 (local address 127.3.85.130:39593)
0619 18:40:48.377338 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:48.377344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:48.377430 (+ 86us) server_negotiation.cc:407] Connection header received
0619 18:40:48.377647 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:48.377652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:48.377715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:48.377811 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:48.378165 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.378958 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.380802 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.381210 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.381360 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:48.381729 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:48.381847 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:48.381944 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:48.382081 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 18:40:48.382151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:40:48.382515 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:48.377104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.3.85.130:39253)
0619 18:40:48.377319 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:48.377339 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:48.377521 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:48.377821 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:48.377826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:48.377836 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:40:48.378016 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:48.378025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.379105 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.379114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:48.380635 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:48.380650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:48.381378 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:48.381384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:48.381538 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:48.381575 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:40:48.381942 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:40:48.381953 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:40:48.382055 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:40:48.382193 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:40:48.392864 5322 raft_consensus.cc:1261] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Refusing update from remote peer cbfe69b9bad447e7955cb8e6ced37746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:48.392935 5604 raft_consensus.cc:1261] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Refusing update from remote peer cbfe69b9bad447e7955cb8e6ced37746: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:48.393493 5678 consensus_queue.cc:1035] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:48.393828 5716 consensus_queue.cc:1035] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:40:48.419778 5700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:40:50.033591 5163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:50.033895 5163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:50.038758 5163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Sending DeleteTablet for 3 replicas of tablet c780dc1539cf4fcd95065339682c15fd
I20260619 18:40:50.039746 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet c780dc1539cf4fcd95065339682c15fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39424
I20260619 18:40:50.039891 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet c780dc1539cf4fcd95065339682c15fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:41470
I20260619 18:40:50.039914 5580 tablet_service.cc:1514] Processing DeleteTablet for tablet c780dc1539cf4fcd95065339682c15fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:48076
I20260619 18:40:50.040630 5759 tablet_replica.cc:330] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: stopping tablet replica
I20260619 18:40:50.040874 5758 tablet_replica.cc:330] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: stopping tablet replica
I20260619 18:40:50.040920 5759 raft_consensus.cc:2229] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:50.041266 5758 raft_consensus.cc:2229] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:50.041500 5759 raft_consensus.cc:2258] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.041689 5758 raft_consensus.cc:2258] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.042810 5759 ts_tablet_manager.cc:1905] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:50.043048 5758 ts_tablet_manager.cc:1905] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:50.046142 5760 tablet_replica.cc:330] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: stopping tablet replica
I20260619 18:40:50.046486 5760 raft_consensus.cc:2229] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:50.046828 5760 raft_consensus.cc:2258] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.047839 5760 ts_tablet_manager.cc:1905] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:40:50 dist-test-slave-09fp krb5kdc[5120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260619 18:40:50.109776 5759 ts_tablet_manager.cc:1918] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:40:50.109843 5760 ts_tablet_manager.cc:1918] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:40:50.109952 5759 log.cc:1196] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/c780dc1539cf4fcd95065339682c15fd
I20260619 18:40:50.110041 5760 log.cc:1196] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/c780dc1539cf4fcd95065339682c15fd
I20260619 18:40:50.110540 5760 ts_tablet_manager.cc:1939] T c780dc1539cf4fcd95065339682c15fd P a0226b6ff32241a89481d01bbbc2d5ac: Deleting consensus metadata
I20260619 18:40:50.110584 5759 ts_tablet_manager.cc:1939] T c780dc1539cf4fcd95065339682c15fd P cbfe69b9bad447e7955cb8e6ced37746: Deleting consensus metadata
I20260619 18:40:50.110630 5758 ts_tablet_manager.cc:1918] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:40:50.110773 5758 log.cc:1196] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/c780dc1539cf4fcd95065339682c15fd
I20260619 18:40:50.111261 5758 ts_tablet_manager.cc:1939] T c780dc1539cf4fcd95065339682c15fd P 4a70499823a3405baaa10407fb9636ae: Deleting consensus metadata
Jun 19 18:40:50 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:50.129522 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:50.113133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.3.85.190:39101)
0619 18:40:50.113953 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:50.113960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:50.113989 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:40:50.114065 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:50.114070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:50.114141 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:50.114263 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:50.115710 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:50.116546 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:50.117865 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:50.118258 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:50.118420 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:50.123988 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:50.123995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:50.123999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:50.124041 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:50.126833 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:50.127557 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:50.127561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:50.127563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:50.127697 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:50.128056 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:50.128060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:50.128062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:50.128359 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:50.128616 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:50.129009 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 18:40:50.129185 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":479,"threads_started":1}
I20260619 18:40:50.133319 5163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:50.133991 5163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:50.141258 5302 tablet_service.cc:1467] Processing CreateTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 (DEFAULT_TABLE table=test-table [id=50dce513346d49c08b77aa92b1bf78c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.141258 5442 tablet_service.cc:1467] Processing CreateTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 (DEFAULT_TABLE table=test-table [id=50dce513346d49c08b77aa92b1bf78c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.141453 5580 tablet_service.cc:1467] Processing CreateTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 (DEFAULT_TABLE table=test-table [id=50dce513346d49c08b77aa92b1bf78c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.141719 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4af27b20b14aa4af756d97dff95ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.141721 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4af27b20b14aa4af756d97dff95ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.141925 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb4af27b20b14aa4af756d97dff95ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.148257 5769 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Bootstrap starting.
I20260619 18:40:50.149063 5771 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Bootstrap starting.
I20260619 18:40:50.149888 5770 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Bootstrap starting.
I20260619 18:40:50.150867 5771 tablet_bootstrap.cc:654] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.151648 5769 tablet_bootstrap.cc:654] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.151778 5770 tablet_bootstrap.cc:654] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.153728 5771 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: No bootstrap required, opened a new log
I20260619 18:40:50.153908 5771 ts_tablet_manager.cc:1397] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:40:50.154277 5769 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: No bootstrap required, opened a new log
I20260619 18:40:50.154495 5769 ts_tablet_manager.cc:1397] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:40:50.154567 5770 tablet_bootstrap.cc:492] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: No bootstrap required, opened a new log
I20260619 18:40:50.154716 5770 ts_tablet_manager.cc:1397] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:40:50.154808 5771 raft_consensus.cc:348] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.155059 5771 raft_consensus.cc:374] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.155148 5771 raft_consensus.cc:729] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.155332 5769 raft_consensus.cc:348] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.155418 5771 consensus_queue.cc:260] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.155591 5769 raft_consensus.cc:374] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.155689 5769 raft_consensus.cc:729] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.155539 5770 raft_consensus.cc:348] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.155776 5770 raft_consensus.cc:374] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.155867 5770 raft_consensus.cc:729] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a70499823a3405baaa10407fb9636ae, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.155906 5769 consensus_queue.cc:260] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.156111 5770 consensus_queue.cc:260] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.157012 5769 ts_tablet_manager.cc:1428] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:40:50.157596 5771 ts_tablet_manager.cc:1428] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:40:50.158296 5770 ts_tablet_manager.cc:1428] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:50.184821 5149 catalog_manager.cc:4841] TS cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593): tablet c780dc1539cf4fcd95065339682c15fd (table test-table [id=abe8bbcf69d1442bad158f95c7a9980b]) successfully deleted
I20260619 18:40:50.185077 5150 catalog_manager.cc:4841] TS 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665): tablet c780dc1539cf4fcd95065339682c15fd (table test-table [id=abe8bbcf69d1442bad158f95c7a9980b]) successfully deleted
I20260619 18:40:50.184861 5152 catalog_manager.cc:4841] TS a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317): tablet c780dc1539cf4fcd95065339682c15fd (table test-table [id=abe8bbcf69d1442bad158f95c7a9980b]) successfully deleted
I20260619 18:40:50.211213 5775 raft_consensus.cc:482] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:50.211470 5775 raft_consensus.cc:504] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.212265 5775 leader_election.cc:288] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593)
I20260619 18:40:50.212859 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4af27b20b14aa4af756d97dff95ce2" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae" is_pre_election: true
I20260619 18:40:50.213151 5322 raft_consensus.cc:2454] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:50.213346 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4af27b20b14aa4af756d97dff95ce2" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746" is_pre_election: true
I20260619 18:40:50.213630 5462 raft_consensus.cc:2454] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:50.213703 5533 leader_election.cc:302] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, a0226b6ff32241a89481d01bbbc2d5ac; no voters:
I20260619 18:40:50.214087 5775 raft_consensus.cc:2781] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:50.214218 5775 raft_consensus.cc:482] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:50.214323 5775 raft_consensus.cc:3037] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.304960 5775 raft_consensus.cc:504] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.305702 5775 leader_election.cc:288] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Requested vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593)
I20260619 18:40:50.306311 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4af27b20b14aa4af756d97dff95ce2" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746"
I20260619 18:40:50.306347 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb4af27b20b14aa4af756d97dff95ce2" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae"
I20260619 18:40:50.306612 5462 raft_consensus.cc:3037] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.306689 5322 raft_consensus.cc:3037] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.404958 5462 raft_consensus.cc:2454] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:50.404950 5322 raft_consensus.cc:2454] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:50.405647 5533 leader_election.cc:302] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, a0226b6ff32241a89481d01bbbc2d5ac; no voters:
I20260619 18:40:50.406044 5775 raft_consensus.cc:2781] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:50.406669 5775 raft_consensus.cc:686] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Becoming Leader. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Running, Role: LEADER
I20260619 18:40:50.407045 5775 consensus_queue.cc:237] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.410245 5164 catalog_manager.cc:5495] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131). New cstate: current_term: 1 leader_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } health_report { overall_health: HEALTHY } } }
I20260619 18:40:50.464733 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:50.458733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.3.85.131:42317)
0619 18:40:50.459888 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:50.459892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:50.459912 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:40:50.459972 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:50.459976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:50.460039 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:50.460131 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:50.460642 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:50.461501 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:50.462990 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:50.463461 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:50.463577 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:50.464051 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:50.464170 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:50.464241 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:50.464387 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:40:50.464452 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":879,"threads_started":1}
I20260619 18:40:50.475117 5322 raft_consensus.cc:1261] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Refusing update from remote peer a0226b6ff32241a89481d01bbbc2d5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:50.475117 5462 raft_consensus.cc:1261] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Refusing update from remote peer a0226b6ff32241a89481d01bbbc2d5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:50.475802 5777 consensus_queue.cc:1035] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:50.476159 5775 consensus_queue.cc:1035] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:50.530799 5165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:50.531010 5165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:50.533969 5165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Sending DeleteTablet for 3 replicas of tablet bb4af27b20b14aa4af756d97dff95ce2
I20260619 18:40:50.534893 5580 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:48076
I20260619 18:40:50.534914 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39424
I20260619 18:40:50.534906 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet bb4af27b20b14aa4af756d97dff95ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:50 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:41470
I20260619 18:40:50.535187 5759 tablet_replica.cc:330] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: stopping tablet replica
I20260619 18:40:50.535190 5760 tablet_replica.cc:330] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: stopping tablet replica
I20260619 18:40:50.535190 5758 tablet_replica.cc:330] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: stopping tablet replica
I20260619 18:40:50.535387 5760 raft_consensus.cc:2229] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:50.535383 5758 raft_consensus.cc:2229] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:50.535390 5759 raft_consensus.cc:2229] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:50.535576 5758 raft_consensus.cc:2258] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.535588 5759 raft_consensus.cc:2258] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.535706 5760 raft_consensus.cc:2258] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:50.536072 5165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:40:50.536648 5758 ts_tablet_manager.cc:1905] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:40:50.536814 5165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:50.537135 5759 ts_tablet_manager.cc:1905] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:50.537416 5760 ts_tablet_manager.cc:1905] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:50.544871 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 (DEFAULT_TABLE table=test-table [id=d17dc24ec2a7452e8c1702335adf605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.544893 5580 tablet_service.cc:1467] Processing CreateTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 (DEFAULT_TABLE table=test-table [id=d17dc24ec2a7452e8c1702335adf605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.545002 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 (DEFAULT_TABLE table=test-table [id=d17dc24ec2a7452e8c1702335adf605d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:50.605391 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39df06ef8cce45c89ba0e1e5efabca61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.605422 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39df06ef8cce45c89ba0e1e5efabca61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.609702 5760 ts_tablet_manager.cc:1918] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:50.609989 5760 log.cc:1196] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/bb4af27b20b14aa4af756d97dff95ce2
I20260619 18:40:50.610606 5760 ts_tablet_manager.cc:1939] T bb4af27b20b14aa4af756d97dff95ce2 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting consensus metadata
I20260619 18:40:50.611240 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39df06ef8cce45c89ba0e1e5efabca61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:50.613411 5769 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Bootstrap starting.
I20260619 18:40:50.615197 5769 tablet_bootstrap.cc:654] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.618196 5770 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: Bootstrap starting.
I20260619 18:40:50.620220 5770 tablet_bootstrap.cc:654] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.621271 5771 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Bootstrap starting.
I20260619 18:40:50.622134 5769 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: No bootstrap required, opened a new log
I20260619 18:40:50.622336 5769 ts_tablet_manager.cc:1397] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260619 18:40:50.622373 5770 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: No bootstrap required, opened a new log
I20260619 18:40:50.622556 5770 ts_tablet_manager.cc:1397] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 18:40:50.622951 5771 tablet_bootstrap.cc:654] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:50.623301 5769 raft_consensus.cc:348] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.623286 5770 raft_consensus.cc:348] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.623584 5770 raft_consensus.cc:374] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.623586 5769 raft_consensus.cc:374] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.623670 5769 raft_consensus.cc:729] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.623672 5770 raft_consensus.cc:729] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a70499823a3405baaa10407fb9636ae, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.623922 5769 consensus_queue.cc:260] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.623922 5770 consensus_queue.cc:260] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.624509 5769 ts_tablet_manager.cc:1428] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:50.625066 5770 ts_tablet_manager.cc:1428] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:50.626865 5771 tablet_bootstrap.cc:492] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: No bootstrap required, opened a new log
I20260619 18:40:50.627130 5771 ts_tablet_manager.cc:1397] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:40:50.627997 5771 raft_consensus.cc:348] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.628257 5771 raft_consensus.cc:374] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:50.628340 5771 raft_consensus.cc:729] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfe69b9bad447e7955cb8e6ced37746, State: Initialized, Role: FOLLOWER
I20260619 18:40:50.628589 5771 consensus_queue.cc:260] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.629374 5771 ts_tablet_manager.cc:1428] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:50.668318 5152 catalog_manager.cc:4841] TS a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317): tablet bb4af27b20b14aa4af756d97dff95ce2 (table test-table [id=50dce513346d49c08b77aa92b1bf78c3]) successfully deleted
I20260619 18:40:50.709429 5758 ts_tablet_manager.cc:1918] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:50.709667 5758 log.cc:1196] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/bb4af27b20b14aa4af756d97dff95ce2
I20260619 18:40:50.709606 5759 ts_tablet_manager.cc:1918] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:40:50.709985 5759 log.cc:1196] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/bb4af27b20b14aa4af756d97dff95ce2
I20260619 18:40:50.710166 5758 ts_tablet_manager.cc:1939] T bb4af27b20b14aa4af756d97dff95ce2 P 4a70499823a3405baaa10407fb9636ae: Deleting consensus metadata
I20260619 18:40:50.711118 5759 ts_tablet_manager.cc:1939] T bb4af27b20b14aa4af756d97dff95ce2 P cbfe69b9bad447e7955cb8e6ced37746: Deleting consensus metadata
I20260619 18:40:50.713285 5149 catalog_manager.cc:4841] TS cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593): tablet bb4af27b20b14aa4af756d97dff95ce2 (table test-table [id=50dce513346d49c08b77aa92b1bf78c3]) successfully deleted
I20260619 18:40:50.714186 5150 catalog_manager.cc:4841] TS 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665): tablet bb4af27b20b14aa4af756d97dff95ce2 (table test-table [id=50dce513346d49c08b77aa92b1bf78c3]) successfully deleted
I20260619 18:40:50.850841 5775 raft_consensus.cc:482] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:50.851089 5775 raft_consensus.cc:504] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.851783 5775 leader_election.cc:288] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593)
I20260619 18:40:50.852402 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39df06ef8cce45c89ba0e1e5efabca61" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae" is_pre_election: true
I20260619 18:40:50.852396 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39df06ef8cce45c89ba0e1e5efabca61" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746" is_pre_election: true
I20260619 18:40:50.852674 5322 raft_consensus.cc:2454] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:50.852674 5462 raft_consensus.cc:2454] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 0.
I20260619 18:40:50.853116 5533 leader_election.cc:302] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a70499823a3405baaa10407fb9636ae, a0226b6ff32241a89481d01bbbc2d5ac; no voters:
I20260619 18:40:50.853430 5775 raft_consensus.cc:2781] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:50.853533 5775 raft_consensus.cc:482] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:50.853600 5775 raft_consensus.cc:3037] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.921358 5775 raft_consensus.cc:504] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.922129 5775 leader_election.cc:288] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Requested vote from peers 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665), cbfe69b9bad447e7955cb8e6ced37746 (127.3.85.130:39593)
I20260619 18:40:50.922821 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39df06ef8cce45c89ba0e1e5efabca61" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbfe69b9bad447e7955cb8e6ced37746"
I20260619 18:40:50.922804 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39df06ef8cce45c89ba0e1e5efabca61" candidate_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a70499823a3405baaa10407fb9636ae"
I20260619 18:40:50.923096 5322 raft_consensus.cc:3037] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.923100 5462 raft_consensus.cc:3037] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:50.925891 5462 raft_consensus.cc:2454] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:50.926478 5532 leader_election.cc:302] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0226b6ff32241a89481d01bbbc2d5ac, cbfe69b9bad447e7955cb8e6ced37746; no voters:
I20260619 18:40:50.926851 5775 raft_consensus.cc:2781] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:50.926970 5775 raft_consensus.cc:686] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Becoming Leader. State: Replica: a0226b6ff32241a89481d01bbbc2d5ac, State: Running, Role: LEADER
I20260619 18:40:50.927326 5775 consensus_queue.cc:237] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } }
I20260619 18:40:50.930244 5165 catalog_manager.cc:5495] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131). New cstate: current_term: 1 leader_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0226b6ff32241a89481d01bbbc2d5ac" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42317 } health_report { overall_health: HEALTHY } } }
I20260619 18:40:51.022100 5322 raft_consensus.cc:2454] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0226b6ff32241a89481d01bbbc2d5ac in term 1.
I20260619 18:40:51.040036 5462 raft_consensus.cc:1261] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Refusing update from remote peer a0226b6ff32241a89481d01bbbc2d5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:51.040036 5321 raft_consensus.cc:1261] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Refusing update from remote peer a0226b6ff32241a89481d01bbbc2d5ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:51.040622 5775 consensus_queue.cc:1035] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a70499823a3405baaa10407fb9636ae" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:51.040908 5784 consensus_queue.cc:1035] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbfe69b9bad447e7955cb8e6ced37746" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:51.047113 5785 mvcc.cc:205] Tried to move back new op lower bound from 7298639671455133696 to 7298639671000772608. Current Snapshot: MvccSnapshot[applied={T|T < 7298639671455133696}]
W20260619 18:40:51.068948 5742 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:40:52.681738 5164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:52.681974 5164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:52.687520 5164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4c231c9a7c544c8935cf3f23796be91: Sending DeleteTablet for 3 replicas of tablet 39df06ef8cce45c89ba0e1e5efabca61
I20260619 18:40:52.688475 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:52 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39424
I20260619 18:40:52.688499 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:52 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:41470
I20260619 18:40:52.689165 5580 tablet_service.cc:1514] Processing DeleteTablet for tablet 39df06ef8cce45c89ba0e1e5efabca61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:52 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:48076
I20260619 18:40:52.689749 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5236
I20260619 18:40:52.691454 5820 tablet_replica.cc:330] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: stopping tablet replica
I20260619 18:40:52.691681 5820 raft_consensus.cc:2229] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:52.691954 5820 raft_consensus.cc:2258] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:52.692698 5819 tablet_replica.cc:330] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: stopping tablet replica
I20260619 18:40:52.692922 5819 raft_consensus.cc:2229] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:52.693229 5819 raft_consensus.cc:2258] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:52.693727 5820 ts_tablet_manager.cc:1905] T 39df06ef8cce45c89ba0e1e5efabca61 P 4a70499823a3405baaa10407fb9636ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:52.693831 5821 tablet_replica.cc:330] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: stopping tablet replica
I20260619 18:40:52.694593 5819 ts_tablet_manager.cc:1905] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:52.694712 5821 raft_consensus.cc:2229] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:52.695182 5821 raft_consensus.cc:2258] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:52.696266 5821 ts_tablet_manager.cc:1905] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:40:52 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:40:52.718890 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.691257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.3.85.129:46665)
0619 18:40:52.693441 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:52.693450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:52.693478 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:40:52.693567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:52.693576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:52.693666 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:52.693821 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:52.695463 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.696651 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.698585 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.699095 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.699241 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:52.708478 (+ 9237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:52.708503 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:52.708529 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:52.708587 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:52.714582 (+ 5995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.715680 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.715695 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.715697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.715862 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.716394 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.716400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.716403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.717129 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:52.717487 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:52.717854 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 18:40:52.718281 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2026,"thread_start_us":964,"threads_started":1}
W20260619 18:40:52.719959 5367 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33654
W20260619 18:40:52.720819 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5236: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:52.721282 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5236
I20260619 18:40:52.793172 5819 ts_tablet_manager.cc:1918] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:40:52.793154 5821 ts_tablet_manager.cc:1918] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:40:52.793321 5821 log.cc:1196] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/39df06ef8cce45c89ba0e1e5efabca61
I20260619 18:40:52.793498 5819 log.cc:1196] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/39df06ef8cce45c89ba0e1e5efabca61
I20260619 18:40:52.793797 5821 ts_tablet_manager.cc:1939] T 39df06ef8cce45c89ba0e1e5efabca61 P a0226b6ff32241a89481d01bbbc2d5ac: Deleting consensus metadata
I20260619 18:40:52.794075 5819 ts_tablet_manager.cc:1939] T 39df06ef8cce45c89ba0e1e5efabca61 P cbfe69b9bad447e7955cb8e6ced37746: Deleting consensus metadata
W20260619 18:40:52.807070 5150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:46665 (error 108)
I20260619 18:40:52.807348 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5376
I20260619 18:40:52.809756 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.808700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.0.0.1:33670)
0619 18:40:52.809259 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.809439 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
W20260619 18:40:52.810055 5150 catalog_manager.cc:4584] TS 4a70499823a3405baaa10407fb9636ae (127.3.85.129:46665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39df06ef8cce45c89ba0e1e5efabca61: Network error: Client connection negotiation failed: client connection to 127.3.85.129:46665: connect: Connection refused (error 111)
Jun 19 18:40:52 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:40:52.826253 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.808237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.3.85.130:39593)
0619 18:40:52.808757 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:52.808762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:52.808780 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:40:52.809449 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:52.809454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:52.809570 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:52.809716 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:52.811344 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.812235 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.813708 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.814132 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.814277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:52.818993 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:52.819010 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:52.819026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:52.819070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:52.822353 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.823304 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.823314 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.823316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.823524 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.823970 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.823977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.823981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.824387 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:52.824716 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:52.825548 (+ 832us) server_negotiation.cc:299] Negotiation successful
0619 18:40:52.825814 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
W20260619 18:40:52.827088 5507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57334
W20260619 18:40:52.827814 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:52.828006 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5376
I20260619 18:40:52.843096 5152 catalog_manager.cc:4841] TS a0226b6ff32241a89481d01bbbc2d5ac (127.3.85.131:42317): tablet 39df06ef8cce45c89ba0e1e5efabca61 (table test-table [id=d17dc24ec2a7452e8c1702335adf605d]) successfully deleted
I20260619 18:40:52.863727 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.863127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.0.0.1:33676)
0619 18:40:52.863375 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.863499 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:40:52.864974 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.864446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:46665 (local address 127.0.0.1:33682)
0619 18:40:52.864630 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.864760 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:40:52.867439 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5516
I20260619 18:40:52.867877 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.867461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:57346)
0619 18:40:52.867622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.867720 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.130:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Jun 19 18:40:52 dist-test-slave-09fp krb5kdc[5120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:40:52.884126 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.868288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.3.85.131:42317)
0619 18:40:52.868842 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:52.868847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:52.868863 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:40:52.868914 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:52.868918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:52.868981 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:52.869086 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:52.870361 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.871305 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.872538 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:52.872948 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:52.873057 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:52.877532 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:52.877549 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:52.877566 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:52.877608 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:52.881348 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.882126 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.882135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.882138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.882277 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:52.882684 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:52.882687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:52.882689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:52.883043 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:52.883268 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:52.883551 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 18:40:52.883732 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
W20260619 18:40:52.884984 5646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56550
W20260619 18:40:52.885805 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5516: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:40:52.886018 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5516
I20260619 18:40:52.900537 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.899878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:57358)
0619 18:40:52.900188 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.900311 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.130:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:40:52.901913 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:52.901417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:57364)
0619 18:40:52.901643 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:52.901744 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.130:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:40:52.910729 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5133
I20260619 18:40:52.912624 5205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:53.032191 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.031373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39593 (local address 127.0.0.1:57370)
0619 18:40:53.031815 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.031948 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.130:39593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316}
I20260619 18:40:53.032462 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5133
2026-06-19T18:40:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9747 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:53 dist-test-slave-09fp krb5kdc[5832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:53 dist-test-slave-09fp krb5kdc[5832](info): set up 2 sockets
Jun 19 18:40:53 dist-test-slave-09fp krb5kdc[5832](info): commencing operation
krb5kdc: starting...
W20260619 18:40:54.667168 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.584s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:54 dist-test-slave-09fp krb5kdc[5832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:54.873962 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:42527
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:42527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:55.241314 5845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:55.241693 5845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:55.241853 5845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:55.249886 5845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:55.250061 5845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:55.250146 5845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:55.250249 5845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:55.250315 5845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:55.261335 5845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:42527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:42527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:55.262939 5845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:55.264381 5845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:55.277242 5850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:55.277313 5853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:55.277282 5851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:55.277938 5845 server_base.cc:1034] running on GCE node
I20260619 18:40:55.278880 5845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:55.280458 5845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:55.281719 5845 hybrid_clock.cc:648] HybridClock initialized: now 1781894455281667 us; error 74 us; skew 500 ppm
I20260619 18:40:55.282087 5845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:55 dist-test-slave-09fp krb5kdc[5832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:55.285562 5845 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:55.287364 5845 webserver.cc:460] Webserver started at http://127.3.85.190:35507/ using document root <none> and password file <none>
I20260619 18:40:55.288107 5845 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:55.288218 5845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:55.288506 5845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:55.291817 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "3dcf3c2f92464f8b94441fbb1f0d4386"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "d681acb0f9c7a393d60f97303acc63b7"
server_key_iv: "8758f336ac6b70e39f16eac2239785a3"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.292663 5845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "3dcf3c2f92464f8b94441fbb1f0d4386"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "d681acb0f9c7a393d60f97303acc63b7"
server_key_iv: "8758f336ac6b70e39f16eac2239785a3"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.298581 5845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:40:55.302650 5860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:55.304273 5845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:40:55.304482 5845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "3dcf3c2f92464f8b94441fbb1f0d4386"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "d681acb0f9c7a393d60f97303acc63b7"
server_key_iv: "8758f336ac6b70e39f16eac2239785a3"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.304702 5845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:55.332293 5845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:55.336673 5845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:55.336997 5845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:55.356758 5845 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:42527
I20260619 18:40:55.356781 5911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:42527 every 8 connection(s)
I20260619 18:40:55.358285 5845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:55.362286 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:55.364874 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5845
I20260619 18:40:55.365113 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:55.365914 3414 external_mini_cluster.cc:1428] Setting key fcab869ad3ed89b9fc25bd1a10e6499d
I20260619 18:40:55.375929 5912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386: Bootstrap starting.
I20260619 18:40:55.380312 5912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386: Neither blocks nor log segments found. Creating new log.
Jun 19 18:40:55 dist-test-slave-09fp krb5kdc[5832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:55.381661 5912 log.cc:824] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:55.384817 5912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386: No bootstrap required, opened a new log
I20260619 18:40:55.390360 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:55.368231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.3.85.190:42527)
0619 18:40:55.369088 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:55.369100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:55.369145 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:40:55.370898 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:55.370925 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:55.371334 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:55.371720 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:55.373597 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:55.375141 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:55.376930 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:55.377647 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:55.377798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:55.382719 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:55.382728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:55.382745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:55.382795 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:55.385911 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:55.386791 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:55.386798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:55.386805 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:55.386996 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:55.387383 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:55.387389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:55.387392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:55.387963 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:55.388304 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:55.388759 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 18:40:55.389236 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":394,"threads_started":1}
I20260619 18:40:55.393389 5912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } }
I20260619 18:40:55.393874 5912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:55.393997 5912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dcf3c2f92464f8b94441fbb1f0d4386, State: Initialized, Role: FOLLOWER
I20260619 18:40:55.394681 5912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } }
I20260619 18:40:55.394930 5912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:55.395071 5912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:55.395246 5912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:55.397387 5912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } }
I20260619 18:40:55.397950 5912 leader_election.cc:302] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dcf3c2f92464f8b94441fbb1f0d4386; no voters:
I20260619 18:40:55.398612 5912 leader_election.cc:288] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:55.398741 5917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:55.400151 5917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [term 1 LEADER]: Becoming Leader. State: Replica: 3dcf3c2f92464f8b94441fbb1f0d4386, State: Running, Role: LEADER
I20260619 18:40:55.400705 5917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } }
I20260619 18:40:55.401109 5912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:55.403949 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } } }
I20260619 18:40:55.404014 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dcf3c2f92464f8b94441fbb1f0d4386. Latest consensus state: current_term: 1 leader_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcf3c2f92464f8b94441fbb1f0d4386" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 42527 } } }
I20260619 18:40:55.404546 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:55.404548 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:55.405532 5926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:55.412075 5926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:55.422871 5926 catalog_manager.cc:1347] Generated new cluster ID: 6cf4e964ac2b47acb05893d0a4d6dc48
I20260619 18:40:55.423035 5926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:55.442546 5926 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:55.443945 5926 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:55.457401 5926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dcf3c2f92464f8b94441fbb1f0d4386: Generated new TSK 0
I20260619 18:40:55.458308 5926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:40:55.518355 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:55.874337 5938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:55.874732 5938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:55.874882 5938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:55.882555 5938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:55.882680 5938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:55.882884 5938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:40:55.893282 5938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:55.894726 5938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:55.896102 5938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:55.910503 5946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:55.910496 5944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:55.910534 5943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:55.911553 5938 server_base.cc:1034] running on GCE node
I20260619 18:40:55.912259 5938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:55.913324 5938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:55.914602 5938 hybrid_clock.cc:648] HybridClock initialized: now 1781894455914531 us; error 69 us; skew 500 ppm
I20260619 18:40:55.914978 5938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:55 dist-test-slave-09fp krb5kdc[5832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:55.918277 5938 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:40:55.920001 5938 webserver.cc:460] Webserver started at http://127.3.85.129:42299/ using document root <none> and password file <none>
I20260619 18:40:55.920765 5938 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:55.920912 5938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:55.921244 5938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:55.924399 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "44c1396448e24388b8f65f558ec05e15"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "975c1c6aae3e75bdf6f2c79cd09309a8"
server_key_iv: "676aeaa00ae990b1228193243411aa50"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.925261 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "44c1396448e24388b8f65f558ec05e15"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "975c1c6aae3e75bdf6f2c79cd09309a8"
server_key_iv: "676aeaa00ae990b1228193243411aa50"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.931082 5938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:40:55.934973 5953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:55.936674 5938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:40:55.936906 5938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "44c1396448e24388b8f65f558ec05e15"
format_stamp: "Formatted at 2026-06-19 18:40:55 on dist-test-slave-09fp"
server_key: "975c1c6aae3e75bdf6f2c79cd09309a8"
server_key_iv: "676aeaa00ae990b1228193243411aa50"
server_key_version: "encryptionkey@0"
I20260619 18:40:55.937160 5938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:55.952564 5938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:55.957353 5938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:55.957667 5938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:55.958765 5938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:55.960263 5938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:55.960368 5938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:55.960515 5938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:55.960587 5938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:55.995038 5938 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:43813
I20260619 18:40:55.995085 6065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:43813 every 8 connection(s)
I20260619 18:40:55.996448 5938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:56.000821 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 5938
I20260619 18:40:56.001077 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:56.001978 3414 external_mini_cluster.cc:1428] Setting key bd76364084145f97dcd8edb6fab92382
Jun 19 18:40:56 dist-test-slave-09fp krb5kdc[5832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:56.018945 6068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:55.999278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:52445 (local address 127.3.85.190:42527)
0619 18:40:55.999749 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:55.999756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:56.001065 (+ 1309us) server_negotiation.cc:407] Connection header received
0619 18:40:56.003652 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:56.003658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:56.003736 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:56.003878 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:56.005759 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.006604 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.008335 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.008852 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.009018 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:56.012799 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:56.012804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:56.012808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:56.012845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:56.015634 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:56.016498 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:56.016504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:56.016508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:56.016643 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:56.017096 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:56.017102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:56.017106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:56.017389 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:56.017552 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:56.018287 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 18:40:56.018572 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:40:56.020284 6069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:55.999826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:42527 (local address 127.3.85.129:52445)
0619 18:40:56.000861 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:56.000925 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:56.003343 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:56.004227 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:56.004237 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:56.004875 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:56.005566 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:56.005587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.006788 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.006792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:56.008156 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:56.008172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.009111 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.009120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:56.009380 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:56.010150 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:56.010175 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:56.012517 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:56.015861 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:56.015875 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:56.015892 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:56.016359 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:56.016769 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:56.016775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:56.016779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:56.016981 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:56.017514 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:56.017525 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:56.017902 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 18:40:56.018181 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 18:40:56.018795 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":387,"threads_started":1}
I20260619 18:40:56.022576 6066 heartbeater.cc:347] Connected to a master server at 127.3.85.190:42527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:40:56.022992 6066 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:56.024008 6066 heartbeater.cc:511] Master 127.3.85.190:42527 requested a full tablet report, sending...
I20260619 18:40:56.026778 5877 ts_manager.cc:194] Registered new tserver with Master: 44c1396448e24388b8f65f558ec05e15 (127.3.85.129:43813)
I20260619 18:40:56.028999 5877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:52445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:40:56.050637 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:56.392892 6072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:56.393210 6072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:56.393308 6072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:56.400954 6072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:56.401088 6072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:56.401270 6072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:40:56.411962 6072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:56.413336 6072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:56.415086 6072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:56.429236 6078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:56.429226 6077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:56.429368 6080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:56.430550 6072 server_base.cc:1034] running on GCE node
I20260619 18:40:56.431336 6072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:56.432379 6072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:56.433641 6072 hybrid_clock.cc:648] HybridClock initialized: now 1781894456433591 us; error 48 us; skew 500 ppm
I20260619 18:40:56.434000 6072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:56 dist-test-slave-09fp krb5kdc[5832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:56.437523 6072 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:40:56.439280 6072 webserver.cc:460] Webserver started at http://127.3.85.130:36075/ using document root <none> and password file <none>
I20260619 18:40:56.440060 6072 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:56.440217 6072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:56.440543 6072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:56.443879 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "bbfc205fd6b64f4db1d383ce6e294551"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "81917af67abf696e649e2822ecd71ad5"
server_key_iv: "6d74fb783a38d9b042c8b479930d9602"
server_key_version: "encryptionkey@0"
I20260619 18:40:56.444775 6072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "bbfc205fd6b64f4db1d383ce6e294551"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "81917af67abf696e649e2822ecd71ad5"
server_key_iv: "6d74fb783a38d9b042c8b479930d9602"
server_key_version: "encryptionkey@0"
I20260619 18:40:56.450649 6072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:40:56.454628 6087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:56.456255 6072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:56.456486 6072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "bbfc205fd6b64f4db1d383ce6e294551"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "81917af67abf696e649e2822ecd71ad5"
server_key_iv: "6d74fb783a38d9b042c8b479930d9602"
server_key_version: "encryptionkey@0"
I20260619 18:40:56.456722 6072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:56.487296 6072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:56.492219 6072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:56.492543 6072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:56.493593 6072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:56.495157 6072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:56.495268 6072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:56.495429 6072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:56.495503 6072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:56.531394 6072 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40161
I20260619 18:40:56.531450 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40161 every 8 connection(s)
I20260619 18:40:56.532753 6072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:56.541465 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6072
I20260619 18:40:56.541661 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:56.542111 3414 external_mini_cluster.cc:1428] Setting key abbb50dc509543444eb40208c6fd30ff
Jun 19 18:40:56 dist-test-slave-09fp krb5kdc[5832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:56.558940 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:56.535743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:57217 (local address 127.3.85.190:42527)
0619 18:40:56.536161 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:56.536166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:56.537304 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 18:40:56.539290 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:56.539295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:56.539403 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:56.539527 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:56.541416 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.542359 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.544509 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.544960 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.545065 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:56.550550 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:56.550559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:56.550564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:56.550619 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:56.554578 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:56.555851 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:56.555857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:56.555860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:56.556028 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:56.556707 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:56.556720 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:56.556722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:56.557026 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:56.557161 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:56.558365 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0619 18:40:56.558657 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260619 18:40:56.560838 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:56.536367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:42527 (local address 127.3.85.130:57217)
0619 18:40:56.537116 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:56.537162 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:56.538962 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:56.539791 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:56.539801 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:56.540381 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:56.541133 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:56.541165 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.542568 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.542574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:56.544326 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:56.544348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:56.545227 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:56.545237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:56.545630 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:56.546821 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:56.546860 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:56.550161 (+ 3301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:56.554839 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:56.554855 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:56.554880 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:56.555644 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:56.556172 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:56.556181 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:56.556188 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:56.556523 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:56.557192 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:56.557205 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:56.557827 (+ 622us) client_negotiation.cc:769] Sending connection context
0619 18:40:56.558231 (+ 404us) client_negotiation.cc:240] Negotiation successful
0619 18:40:56.558977 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":310,"threads_started":1}
I20260619 18:40:56.562881 6200 heartbeater.cc:347] Connected to a master server at 127.3.85.190:42527
I20260619 18:40:56.563264 6200 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:56.564040 6200 heartbeater.cc:511] Master 127.3.85.190:42527 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:40:56.565631 5877 ts_manager.cc:194] Registered new tserver with Master: bbfc205fd6b64f4db1d383ce6e294551 (127.3.85.130:40161)
I20260619 18:40:56.566425 5877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:57217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:40:56.589962 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:56.940398 6206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:56.940706 6206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:56.940857 6206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:56.948714 6206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:56.948856 6206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:56.949040 6206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:40:56.959720 6206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:42527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:56.961462 6206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:56.963030 6206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:56.977108 6214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:56.977142 6212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:56.977169 6211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:56.978223 6206 server_base.cc:1034] running on GCE node
I20260619 18:40:56.978981 6206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:56.980000 6206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:56.981269 6206 hybrid_clock.cc:648] HybridClock initialized: now 1781894456981222 us; error 49 us; skew 500 ppm
I20260619 18:40:56.981647 6206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:56 dist-test-slave-09fp krb5kdc[5832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:56.985189 6206 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:40:56.986969 6206 webserver.cc:460] Webserver started at http://127.3.85.131:33987/ using document root <none> and password file <none>
I20260619 18:40:56.987766 6206 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:56.987937 6206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:56.988273 6206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:56.991685 6206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "cd931467afba4aeeaff744b0142d006d"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "f732593d12d866fc7b7329e0a3435f91"
server_key_iv: "af1baf0fe1285a15c41066bac6898096"
server_key_version: "encryptionkey@0"
I20260619 18:40:56.992758 6206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "cd931467afba4aeeaff744b0142d006d"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "f732593d12d866fc7b7329e0a3435f91"
server_key_iv: "af1baf0fe1285a15c41066bac6898096"
server_key_version: "encryptionkey@0"
I20260619 18:40:56.999099 6206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:40:57.003095 6221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:57.004791 6206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:57.005007 6206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "cd931467afba4aeeaff744b0142d006d"
format_stamp: "Formatted at 2026-06-19 18:40:56 on dist-test-slave-09fp"
server_key: "f732593d12d866fc7b7329e0a3435f91"
server_key_iv: "af1baf0fe1285a15c41066bac6898096"
server_key_version: "encryptionkey@0"
I20260619 18:40:57.005239 6206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:57.032569 6066 heartbeater.cc:503] Master 127.3.85.190:42527 was elected leader, sending a full tablet report...
I20260619 18:40:57.033882 6206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:57.038410 6206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:57.038808 6206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:57.039835 6206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:57.041239 6206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:57.041342 6206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:57.041527 6206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:57.041596 6206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:57.076473 6206 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:36619
I20260619 18:40:57.076574 6333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:36619 every 8 connection(s)
I20260619 18:40:57.077867 6206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:57.082839 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6206
I20260619 18:40:57.083055 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:57.083616 3414 external_mini_cluster.cc:1428] Setting key dd18731738f24cd6515903ca896975bb
Jun 19 18:40:57 dist-test-slave-09fp krb5kdc[5832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:57.100220 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:57.080731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:60449 (local address 127.3.85.190:42527)
0619 18:40:57.081169 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:57.081174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:57.082206 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 18:40:57.085096 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:57.085104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:57.085216 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:57.085303 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:57.087408 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:57.088198 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:57.089846 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:57.090268 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:57.090372 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:57.094010 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:57.094015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:57.094019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:57.094056 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:57.096647 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:57.097435 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:57.097439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:57.097441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:57.097566 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:57.098083 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:57.098088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:57.098091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:57.098402 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:57.098651 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:57.099587 (+ 936us) server_negotiation.cc:299] Negotiation successful
0619 18:40:57.099808 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
I20260619 18:40:57.101229 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:57.081290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:42527 (local address 127.3.85.131:60449)
0619 18:40:57.082037 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:57.082075 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:57.084580 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:57.085647 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:57.085664 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:57.086378 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:57.087188 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:57.087236 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:57.088382 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:57.088387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:57.089698 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:57.089712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:57.090573 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:57.090579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:57.090813 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:57.091479 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:57.091504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:57.093744 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:57.096809 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:57.096816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:57.096832 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:57.097313 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:57.097676 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:57.097681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:57.097684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:57.097954 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:57.098635 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:57.098644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:57.099112 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 18:40:57.099384 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 18:40:57.099982 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":293,"threads_started":1}
I20260619 18:40:57.102842 6334 heartbeater.cc:347] Connected to a master server at 127.3.85.190:42527
I20260619 18:40:57.103180 6334 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:57.104112 6334 heartbeater.cc:511] Master 127.3.85.190:42527 requested a full tablet report, sending...
I20260619 18:40:57.105697 5877 ts_manager.cc:194] Registered new tserver with Master: cd931467afba4aeeaff744b0142d006d (127.3.85.131:36619)
I20260619 18:40:57.106736 5877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:60449
I20260619 18:40:57.112607 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:40:57.144192 6336 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:40:57.141565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.3.85.190:42527)
0619 18:40:57.141748 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:57.141753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:57.142336 (+ 583us) server_negotiation.cc:407] Connection header received
0619 18:40:57.142499 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:57.142504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:57.142579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:57.142716 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:57.143983 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37768: BlockingRecv error: recv got EOF from 127.0.0.1:37768 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:40:57.147652 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5938
W20260619 18:40:57.150735 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5938: Not authorized: Client connection negotiation failed: client connection to 127.3.85.129:43813: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:40:57.150964 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5938
I20260619 18:40:57.166935 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6072
W20260619 18:40:57.169679 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6072: Not authorized: Client connection negotiation failed: client connection to 127.3.85.130:40161: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:40:57.169859 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6072
I20260619 18:40:57.185897 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6206
W20260619 18:40:57.189031 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6206: Not authorized: Client connection negotiation failed: client connection to 127.3.85.131:36619: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:40:57.189195 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6206
I20260619 18:40:57.205054 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 5845
I20260619 18:40:57.206383 5907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:57.308907 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 5845
2026-06-19T18:40:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4268 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:57 dist-test-slave-09fp krb5kdc[6352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:57 dist-test-slave-09fp krb5kdc[6352](info): set up 2 sockets
Jun 19 18:40:57 dist-test-slave-09fp krb5kdc[6352](info): commencing operation
krb5kdc: starting...
W20260619 18:40:58.407064 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:40:58 dist-test-slave-09fp krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:40:58.627287 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34247
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:58.977018 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:58.977396 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:58.977561 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:58.985080 6365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:58.985270 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:58.985375 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:58.985471 6365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:58.985554 6365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:58.995975 6365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:34247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:40:58.997345 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:58.998689 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:59.011221 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:59.011185 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:59.011245 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:59.012256 6365 server_base.cc:1034] running on GCE node
I20260619 18:40:59.013139 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:59.014804 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:59.016063 6365 hybrid_clock.cc:648] HybridClock initialized: now 1781894459016015 us; error 48 us; skew 500 ppm
I20260619 18:40:59.016450 6365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:40:59 dist-test-slave-09fp krb5kdc[6352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:59.024092 6365 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:40:59.025947 6365 webserver.cc:460] Webserver started at http://127.3.85.190:44627/ using document root <none> and password file <none>
I20260619 18:40:59.026839 6365 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:59.026995 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:59.027341 6365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:59.030620 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "4a2e3dda556b4f208a33a33c7a1fff64"
format_stamp: "Formatted at 2026-06-19 18:40:59 on dist-test-slave-09fp"
server_key: "cc7b46350735a074d38a787fbcd43ab8"
server_key_iv: "ecf27044a9f585efb50c3e9435a661db"
server_key_version: "encryptionkey@0"
I20260619 18:40:59.031540 6365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "4a2e3dda556b4f208a33a33c7a1fff64"
format_stamp: "Formatted at 2026-06-19 18:40:59 on dist-test-slave-09fp"
server_key: "cc7b46350735a074d38a787fbcd43ab8"
server_key_iv: "ecf27044a9f585efb50c3e9435a661db"
server_key_version: "encryptionkey@0"
I20260619 18:40:59.037308 6365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:40:59.041263 6380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:59.042887 6365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:40:59.043123 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "4a2e3dda556b4f208a33a33c7a1fff64"
format_stamp: "Formatted at 2026-06-19 18:40:59 on dist-test-slave-09fp"
server_key: "cc7b46350735a074d38a787fbcd43ab8"
server_key_iv: "ecf27044a9f585efb50c3e9435a661db"
server_key_version: "encryptionkey@0"
I20260619 18:40:59.043346 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:59.060931 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:59.065758 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:59.066088 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:59.087056 6365 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:34247
I20260619 18:40:59.087150 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:34247 every 8 connection(s)
I20260619 18:40:59.088606 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:59.092679 6432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:59.097751 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6365
I20260619 18:40:59.098004 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:40:59.098729 3414 external_mini_cluster.cc:1428] Setting key e6516c1f2d1f8a5ef9a0525596fe1092
I20260619 18:40:59.105726 6432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: Bootstrap starting.
I20260619 18:40:59.109907 6432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:59.111280 6432 log.cc:824] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:40:59 dist-test-slave-09fp krb5kdc[6352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:40:59.115092 6432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: No bootstrap required, opened a new log
I20260619 18:40:59.124040 6432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } }
I20260619 18:40:59.124533 6432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:59.124670 6432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2e3dda556b4f208a33a33c7a1fff64, State: Initialized, Role: FOLLOWER
I20260619 18:40:59.125029 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:59.100780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.3.85.190:34247)
0619 18:40:59.101646 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:59.101658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:59.101702 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:40:59.103903 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:59.103933 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:59.104419 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:59.104843 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:59.106897 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:59.108332 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:59.110265 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:59.111125 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:59.111301 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:59.116223 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:59.116232 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:59.116250 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:59.116305 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:59.120124 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:59.120981 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:59.120990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:59.121000 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:59.121209 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:59.121631 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:59.121637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:59.121641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:59.122303 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:59.122685 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:59.123174 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 18:40:59.123661 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":396,"threads_started":1}
I20260619 18:40:59.125378 6432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } }
I20260619 18:40:59.125633 6432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:59.125788 6432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:59.125996 6432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:59.128896 6432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } }
I20260619 18:40:59.129278 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:40:59.129554 6432 leader_election.cc:302] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a2e3dda556b4f208a33a33c7a1fff64; no voters:
I20260619 18:40:59.130167 6432 leader_election.cc:288] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:59.130318 6437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:59.132418 6432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:59.137393 6437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [term 1 LEADER]: Becoming Leader. State: Replica: 4a2e3dda556b4f208a33a33c7a1fff64, State: Running, Role: LEADER
I20260619 18:40:59.138178 6437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } }
I20260619 18:40:59.142024 6438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } } }
I20260619 18:40:59.142419 6438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:59.144232 6439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a2e3dda556b4f208a33a33c7a1fff64. Latest consensus state: current_term: 1 leader_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2e3dda556b4f208a33a33c7a1fff64" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34247 } } }
I20260619 18:40:59.144577 6439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64 [sys.catalog]: This master's current role is: LEADER
W20260619 18:40:59.153362 6452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:40:59.153570 6452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:40:59.153723 6453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:59.156344 6453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:59.166064 6453 catalog_manager.cc:1347] Generated new cluster ID: b3f62beb66cf48b29b791b950276bf48
I20260619 18:40:59.166217 6453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:59.187388 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:59.177033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.3.85.190:34247)
0619 18:40:59.177232 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:59.177238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:59.177918 (+ 680us) server_negotiation.cc:407] Connection header received
0619 18:40:59.178119 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:59.178126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:59.178217 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:59.178340 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:59.179731 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:59.180991 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:59.183097 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:59.183833 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:59.183987 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:59.185361 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:59.185369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:59.185374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:40:59.186324 (+ 950us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:59.186414 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:40:59.186484 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:59.186615 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:59.186820 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 18:40:59.186937 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":75}
I20260619 18:40:59.202819 6453 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:59.204613 6453 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:59.215083 6453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a2e3dda556b4f208a33a33c7a1fff64: Generated new TSK 0
I20260619 18:40:59.216099 6453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:40:59.231805 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6365
I20260619 18:40:59.233222 6427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:40:59.325703 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6365
2026-06-19T18:40:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2011 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:59 dist-test-slave-09fp krb5kdc[6466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:59 dist-test-slave-09fp krb5kdc[6466](info): set up 2 sockets
Jun 19 18:40:59 dist-test-slave-09fp krb5kdc[6466](info): commencing operation
krb5kdc: starting...
W20260619 18:41:00.435230 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:00 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:00.640074 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40545
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:40545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:00.986259 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:00.986676 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:00.986845 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:00.994531 6479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:00.994674 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:00.994781 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:00.994872 6479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:00.994951 6479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:01.005448 6479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:40545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:01.006978 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:01.008404 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:01.022250 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:01.022222 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:01.022357 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:01.023033 6479 server_base.cc:1034] running on GCE node
I20260619 18:41:01.024009 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:01.025666 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:01.026965 6479 hybrid_clock.cc:648] HybridClock initialized: now 1781894461026907 us; error 58 us; skew 500 ppm
I20260619 18:41:01.027397 6479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:01 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894461, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:01.031003 6479 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:01.032773 6479 webserver.cc:460] Webserver started at http://127.3.85.190:35267/ using document root <none> and password file <none>
I20260619 18:41:01.033566 6479 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:01.033712 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:01.034065 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:01.037406 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "42a79e19243c4c618c0a22896c3ebe48"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "e4f9f91c25bc75ac82f77ce599a7c4b7"
server_key_iv: "6fca0e216916b1205e71ae29990ce28f"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.038321 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "42a79e19243c4c618c0a22896c3ebe48"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "e4f9f91c25bc75ac82f77ce599a7c4b7"
server_key_iv: "6fca0e216916b1205e71ae29990ce28f"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.044476 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 18:41:01.048465 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:01.050025 6479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:01.050259 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "42a79e19243c4c618c0a22896c3ebe48"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "e4f9f91c25bc75ac82f77ce599a7c4b7"
server_key_iv: "6fca0e216916b1205e71ae29990ce28f"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.050523 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:01.072763 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:01.077617 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:01.077965 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:01.097405 6479 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:40545
I20260619 18:41:01.097443 6545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:40545 every 8 connection(s)
I20260619 18:41:01.099021 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:01.102344 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6479
I20260619 18:41:01.102593 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:01.103402 3414 external_mini_cluster.cc:1428] Setting key ced3d3360f965f86a8dd56cfb38dee9d
I20260619 18:41:01.103726 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:01.117220 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48: Bootstrap starting.
Jun 19 18:41:01 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:01.121409 6546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:01.122812 6546 log.cc:824] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:01.126714 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48: No bootstrap required, opened a new log
I20260619 18:41:01.129122 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:01.105721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.3.85.190:40545)
0619 18:41:01.106660 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:01.106672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:01.106719 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:41:01.108899 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:01.108928 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:01.109440 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:01.110035 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:01.111598 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.112935 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.114297 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.115227 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.115445 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:01.120166 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:01.120176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:01.120193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:01.120239 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:01.124357 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:01.125360 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:01.125367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:01.125374 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:01.125578 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:01.126058 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:01.126061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:01.126063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:01.126710 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:01.127039 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:01.127537 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 18:41:01.128005 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":363,"threads_started":1}
I20260619 18:41:01.135069 6546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } }
I20260619 18:41:01.135550 6546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:01.135679 6546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a79e19243c4c618c0a22896c3ebe48, State: Initialized, Role: FOLLOWER
I20260619 18:41:01.136318 6546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } }
I20260619 18:41:01.136600 6546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:01.136749 6546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:01.136922 6546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:01.139322 6546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } }
I20260619 18:41:01.139997 6546 leader_election.cc:302] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42a79e19243c4c618c0a22896c3ebe48; no voters:
I20260619 18:41:01.140643 6546 leader_election.cc:288] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:01.140878 6551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:01.142102 6551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [term 1 LEADER]: Becoming Leader. State: Replica: 42a79e19243c4c618c0a22896c3ebe48, State: Running, Role: LEADER
I20260619 18:41:01.142916 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } }
I20260619 18:41:01.143175 6546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:01.146255 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42a79e19243c4c618c0a22896c3ebe48. Latest consensus state: current_term: 1 leader_uuid: "42a79e19243c4c618c0a22896c3ebe48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } } }
I20260619 18:41:01.146346 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42a79e19243c4c618c0a22896c3ebe48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a79e19243c4c618c0a22896c3ebe48" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40545 } } }
I20260619 18:41:01.146768 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:01.146816 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:01.151105 6558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:01.157763 6558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:01.167541 6558 catalog_manager.cc:1347] Generated new cluster ID: d9f5ae32bff8445a9a51579ed847bfc9
I20260619 18:41:01.167692 6558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:01.181392 6558 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:01.183431 6558 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:01.197036 6558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42a79e19243c4c618c0a22896c3ebe48: Generated new TSK 0
I20260619 18:41:01.198318 6558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:41:01.248310 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:01.602370 6572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:01.602782 6572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:01.602963 6572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:01.610796 6572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:01.610925 6572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:01.611115 6572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:01.621587 6572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:01.623081 6572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:01.624511 6572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:01.639648 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:01.639634 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:01.639927 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:01.641185 6572 server_base.cc:1034] running on GCE node
I20260619 18:41:01.641999 6572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:01.643327 6572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:01.644593 6572 hybrid_clock.cc:648] HybridClock initialized: now 1781894461644543 us; error 46 us; skew 500 ppm
I20260619 18:41:01.644971 6572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:01 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894461, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:01.648674 6572 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:41:01.650408 6572 webserver.cc:460] Webserver started at http://127.3.85.129:44639/ using document root <none> and password file <none>
I20260619 18:41:01.651230 6572 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:01.651383 6572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:01.651708 6572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:01.655414 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "5809b09a7cdd45bf8d5f8afbf3668559"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "33a37b9aea85ccd67d90053dcf8efca6"
server_key_iv: "9f8fc0651fe32276865d637107bc2125"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.656747 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "5809b09a7cdd45bf8d5f8afbf3668559"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "33a37b9aea85ccd67d90053dcf8efca6"
server_key_iv: "9f8fc0651fe32276865d637107bc2125"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.664785 6572 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.007s
I20260619 18:41:01.668771 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:01.670349 6572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:41:01.670605 6572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "5809b09a7cdd45bf8d5f8afbf3668559"
format_stamp: "Formatted at 2026-06-19 18:41:01 on dist-test-slave-09fp"
server_key: "33a37b9aea85ccd67d90053dcf8efca6"
server_key_iv: "9f8fc0651fe32276865d637107bc2125"
server_key_version: "encryptionkey@0"
I20260619 18:41:01.670845 6572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:01.692431 6572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:01.697432 6572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:01.697749 6572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:01.698829 6572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:01.700286 6572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:01.700395 6572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:01.700543 6572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:01.700611 6572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:01.735262 6572 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:42031
I20260619 18:41:01.735360 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:42031 every 8 connection(s)
I20260619 18:41:01.736675 6572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:01.741006 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6572
I20260619 18:41:01.741241 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:01.741894 3414 external_mini_cluster.cc:1428] Setting key 198951b0c0afe6fc57ba2f17e5a4d68c
Jun 19 18:41:01 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894461, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:01.759858 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:01.739369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:52171 (local address 127.3.85.190:40545)
0619 18:41:01.739833 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:01.739837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:01.740779 (+ 942us) server_negotiation.cc:407] Connection header received
0619 18:41:01.742982 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:01.742989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:01.743115 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:01.743250 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:01.745175 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.745912 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.747653 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.748061 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.748207 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:01.752682 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:01.752687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:01.752691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:01.752787 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:01.756491 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:01.757378 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:01.757384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:01.757391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:01.757520 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:01.757993 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:01.757998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:01.758001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:01.758274 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:01.758455 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:01.759244 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 18:41:01.759493 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:01.761019 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:01.739846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40545 (local address 127.3.85.129:52171)
0619 18:41:01.740607 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:01.740648 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:01.742622 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:01.743566 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:01.743576 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:01.744270 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:01.744978 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:01.745010 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.746102 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.746107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:01.747479 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:01.747492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:01.748320 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:01.748327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:01.748586 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:01.749312 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:01.749342 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:01.752341 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:01.756685 (+ 4344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:01.756696 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:01.756718 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:01.757221 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:01.757639 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:01.757644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:01.757648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:01.757867 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:01.758420 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:01.758431 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:01.758855 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 18:41:01.759134 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 18:41:01.759714 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":273,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:41:01.762872 6700 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40545
I20260619 18:41:01.763207 6700 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:01.764017 6700 heartbeater.cc:511] Master 127.3.85.190:40545 requested a full tablet report, sending...
I20260619 18:41:01.766392 6511 ts_manager.cc:194] Registered new tserver with Master: 5809b09a7cdd45bf8d5f8afbf3668559 (127.3.85.129:42031)
I20260619 18:41:01.768535 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:52171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:41:01.787025 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:02.135182 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:02.135514 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:02.135617 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:02.143307 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:02.143481 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:02.143679 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:02.154364 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:02.155920 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:02.157277 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:02.171558 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:02.172107 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:02.172163 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:02.172448 6706 server_base.cc:1034] running on GCE node
I20260619 18:41:02.173194 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:02.174484 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:02.175835 6706 hybrid_clock.cc:648] HybridClock initialized: now 1781894462175743 us; error 77 us; skew 500 ppm
I20260619 18:41:02.176187 6706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:02.179790 6706 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:41:02.181365 6706 webserver.cc:460] Webserver started at http://127.3.85.130:41089/ using document root <none> and password file <none>
I20260619 18:41:02.182103 6706 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:02.182222 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:02.182590 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:02.185762 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "ff4069ca4a9e46f890f22ee3dd2549db"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "aff11a82e56c3f9df10947a36a120cb8"
server_key_iv: "361ec958bfc3174437c3dbfcfc6ac9d1"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.186671 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "ff4069ca4a9e46f890f22ee3dd2549db"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "aff11a82e56c3f9df10947a36a120cb8"
server_key_iv: "361ec958bfc3174437c3dbfcfc6ac9d1"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.192493 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:41:02.196491 6721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.198145 6706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:02.198366 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "ff4069ca4a9e46f890f22ee3dd2549db"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "aff11a82e56c3f9df10947a36a120cb8"
server_key_iv: "361ec958bfc3174437c3dbfcfc6ac9d1"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.198659 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:02.220736 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:02.225772 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:02.226140 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:02.227300 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:02.228987 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:02.229085 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.229214 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:02.229292 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.264667 6706 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:41233
I20260619 18:41:02.264717 6833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:41233 every 8 connection(s)
I20260619 18:41:02.266089 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:02.269284 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6706
I20260619 18:41:02.269495 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:02.270053 3414 external_mini_cluster.cc:1428] Setting key 85db30a8cf4615b7db236d8940382692
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:41:02.289671 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.269079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51417 (local address 127.3.85.190:40545)
0619 18:41:02.269883 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:02.269888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:02.270974 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 18:41:02.273320 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:02.273326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:02.273420 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:02.273544 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:02.275454 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.276245 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.277960 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.278411 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.278608 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.282488 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:02.282497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:02.282504 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:02.282551 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:02.286215 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.287153 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.287160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.287163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.287314 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.287775 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.287785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.287787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.288064 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:02.288220 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:02.289007 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 18:41:02.289268 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":347,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
I20260619 18:41:02.290894 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.269649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40545 (local address 127.3.85.130:51417)
0619 18:41:02.270719 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:02.270812 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:02.273004 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:02.273834 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:02.273844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:02.274537 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:02.275253 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:02.275274 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.276425 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.276429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:02.277789 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:02.277804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.278718 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.278725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:02.279001 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.279767 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:02.279798 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:02.282149 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:02.286439 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:02.286480 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:02.286503 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:02.287012 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:02.287433 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:02.287438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:02.287442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:02.287644 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:02.288196 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:02.288205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:02.288627 (+ 422us) client_negotiation.cc:769] Sending connection context
0619 18:41:02.288909 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 18:41:02.289485 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":383,"threads_started":1}
I20260619 18:41:02.292532 6834 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40545
I20260619 18:41:02.292891 6834 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:02.293865 6834 heartbeater.cc:511] Master 127.3.85.190:40545 requested a full tablet report, sending...
I20260619 18:41:02.295698 6511 ts_manager.cc:194] Registered new tserver with Master: ff4069ca4a9e46f890f22ee3dd2549db (127.3.85.130:41233)
I20260619 18:41:02.296967 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:51417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:41:02.317579 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:02.682029 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:02.682341 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:02.682436 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:02.690061 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:02.690196 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:02.690311 6840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:02.700735 6840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:02.702098 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:02.703737 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:02.720834 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:02.720893 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:02.720786 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:02.722188 6840 server_base.cc:1034] running on GCE node
I20260619 18:41:02.722995 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:02.724133 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:02.725402 6840 hybrid_clock.cc:648] HybridClock initialized: now 1781894462725355 us; error 52 us; skew 500 ppm
I20260619 18:41:02.725832 6840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:02.729429 6840 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:41:02.731175 6840 webserver.cc:460] Webserver started at http://127.3.85.131:41289/ using document root <none> and password file <none>
I20260619 18:41:02.731910 6840 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:02.732028 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:02.732295 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:02.735605 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "a51a508b3941419b8db56066377fecef"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "7e66b7da23e27d9b048fca6cd1ec3dd1"
server_key_iv: "4aa4350be212158403341fa6f295e5ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.736536 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "a51a508b3941419b8db56066377fecef"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "7e66b7da23e27d9b048fca6cd1ec3dd1"
server_key_iv: "4aa4350be212158403341fa6f295e5ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.742678 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:41:02.747768 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.749625 6840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:02.749918 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "a51a508b3941419b8db56066377fecef"
format_stamp: "Formatted at 2026-06-19 18:41:02 on dist-test-slave-09fp"
server_key: "7e66b7da23e27d9b048fca6cd1ec3dd1"
server_key_iv: "4aa4350be212158403341fa6f295e5ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:02.750190 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:02.772007 6700 heartbeater.cc:503] Master 127.3.85.190:40545 was elected leader, sending a full tablet report...
I20260619 18:41:02.782285 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:02.787864 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:02.788226 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:02.789343 6840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:02.791122 6840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:02.791239 6840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.791419 6840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:02.791505 6840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:02.828235 6840 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:44401
I20260619 18:41:02.828289 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:44401 every 8 connection(s)
I20260619 18:41:02.829627 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:02.839869 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6840
I20260619 18:41:02.840054 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:02.840536 3414 external_mini_cluster.cc:1428] Setting key 544c9df009c857b12ea5e046fbc617fb
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:02.851898 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.832830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:37917 (local address 127.3.85.190:40545)
0619 18:41:02.833376 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:02.833383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:02.834325 (+ 942us) server_negotiation.cc:407] Connection header received
0619 18:41:02.836303 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:02.836308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:02.836418 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:02.836544 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:02.838169 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.839035 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.840897 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.841476 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.841619 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.845753 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:02.845758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:02.845762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:02.845798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:02.848451 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.849339 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.849345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.849349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.849488 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.849950 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.849958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.849961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.850252 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:02.850426 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:02.851217 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 18:41:02.851464 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":259,"threads_started":1}
I20260619 18:41:02.852936 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.833412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40545 (local address 127.3.85.131:37917)
0619 18:41:02.834134 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:02.834201 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:02.835988 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:02.836807 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:02.836817 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:02.837371 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:02.837975 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:02.837993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.839218 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.839222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:02.840740 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:02.840754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.841736 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.841746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:02.842021 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.842809 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:02.842833 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:02.845492 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:02.848663 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:02.848675 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:02.848700 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:02.849183 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:02.849614 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:02.849619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:02.849623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:02.849836 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:02.850389 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:02.850398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:02.850844 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 18:41:02.851126 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 18:41:02.851703 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":294,"threads_started":1}
I20260619 18:41:02.854574 6968 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40545
I20260619 18:41:02.854931 6968 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:02.855984 6968 heartbeater.cc:511] Master 127.3.85.190:40545 requested a full tablet report, sending...
I20260619 18:41:02.857627 6511 ts_manager.cc:194] Registered new tserver with Master: a51a508b3941419b8db56066377fecef (127.3.85.131:44401)
I20260619 18:41:02.858412 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:37917
I20260619 18:41:02.869724 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:02.922145 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.906619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.3.85.190:40545)
0619 18:41:02.906837 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:02.906848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:02.907305 (+ 457us) server_negotiation.cc:407] Connection header received
0619 18:41:02.907545 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:02.907549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:02.907621 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:02.907734 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:02.909166 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.909924 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.911609 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.912063 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.912232 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.916626 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:02.916632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:02.916635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:02.916677 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:02.919388 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.920141 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.920146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.920148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.920273 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.920668 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.920678 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.920680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.921005 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:02.921254 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:02.921585 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:41:02.921828 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 18:41:02.922755 6511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57404
I20260619 18:41:02.925087 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6572
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:02.949303 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.926905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.3.85.129:42031)
0619 18:41:02.927614 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:02.927621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:02.927649 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:41:02.927715 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:02.927721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:02.927932 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:02.928138 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:02.930351 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.932090 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.934186 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.935172 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.935350 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.940904 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:02.940928 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:02.940953 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:02.941016 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:02.945670 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.946646 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.946660 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.946666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.946870 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.947286 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.947294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.947298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.947836 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:02.948204 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:02.948422 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 18:41:02.948736 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":363,"threads_started":1}
W20260619 18:41:02.950774 6695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39344
W20260619 18:41:02.952024 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:41:02.952256 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6572
I20260619 18:41:02.969696 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6706
Jun 19 18:41:02 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:41:02.987797 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:02.970749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.3.85.130:41233)
0619 18:41:02.971360 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:02.971367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:02.971392 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:02.971472 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:02.971478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:02.971692 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:02.971871 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:02.973146 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.974224 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.975575 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:02.976278 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:02.976434 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:02.981340 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:02.981357 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:02.981374 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:02.981419 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:02.984638 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.985431 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.985439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.985441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.985573 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:02.985989 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:02.985994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:02.985996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:02.986383 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:02.986764 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:02.987093 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 18:41:02.987399 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
W20260619 18:41:02.989087 6829 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47850
W20260619 18:41:02.990316 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:41:02.990504 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6706
I20260619 18:41:03.006711 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6840
Jun 19 18:41:03 dist-test-slave-09fp krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894462, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:41:03.023790 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:03.007709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.3.85.131:44401)
0619 18:41:03.008320 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:03.008327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:03.008357 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:41:03.008420 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:03.008426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:03.008643 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:03.008777 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:03.010115 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:03.011292 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:03.012554 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:03.013127 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:03.013245 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:03.017717 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:03.017732 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:03.017755 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:03.017799 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:03.021188 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:03.021977 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:03.021984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:03.021987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:03.022105 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:03.022425 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:03.022428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:03.022430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:03.022819 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:03.023061 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:03.023300 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 18:41:03.023510 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":296,"threads_started":1}
W20260619 18:41:03.025125 6963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35344
W20260619 18:41:03.026274 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:41:03.026523 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6840
I20260619 18:41:03.042152 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6479
I20260619 18:41:03.043399 6541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:03.145068 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6479
2026-06-19T18:41:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3825 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:03 dist-test-slave-09fp krb5kdc[6986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:03 dist-test-slave-09fp krb5kdc[6986](info): set up 2 sockets
Jun 19 18:41:03 dist-test-slave-09fp krb5kdc[6986](info): commencing operation
krb5kdc: starting...
W20260619 18:41:04.247274 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:04 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:04.458801 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37893
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:04.803251 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:04.803582 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:04.803673 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:04.810997 6999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:04.811121 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:04.811168 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:04.811218 6999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:04.811262 6999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:04.821969 6999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:04.823458 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:04.824895 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:04.837886 7007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:04.837916 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:04.838820 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:04.838842 6999 server_base.cc:1034] running on GCE node
I20260619 18:41:04.839771 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:04.841276 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:04.842559 6999 hybrid_clock.cc:648] HybridClock initialized: now 1781894464842508 us; error 40 us; skew 500 ppm
I20260619 18:41:04.842986 6999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:04 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894464, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:04.851207 6999 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:04.852991 6999 webserver.cc:460] Webserver started at http://127.3.85.190:38743/ using document root <none> and password file <none>
I20260619 18:41:04.853767 6999 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:04.853925 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:04.854262 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:04.857494 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "b1e71c7d2f3b4fa59bd650055abe280b"
format_stamp: "Formatted at 2026-06-19 18:41:04 on dist-test-slave-09fp"
server_key: "68f6551f47b099d2c17ffda13e1dd7bb"
server_key_iv: "b436e67558626ab9d288e26a5fd3ad85"
server_key_version: "encryptionkey@0"
I20260619 18:41:04.858363 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "b1e71c7d2f3b4fa59bd650055abe280b"
format_stamp: "Formatted at 2026-06-19 18:41:04 on dist-test-slave-09fp"
server_key: "68f6551f47b099d2c17ffda13e1dd7bb"
server_key_iv: "b436e67558626ab9d288e26a5fd3ad85"
server_key_version: "encryptionkey@0"
I20260619 18:41:04.863963 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:41:04.867674 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:04.869172 6999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:04.869385 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "b1e71c7d2f3b4fa59bd650055abe280b"
format_stamp: "Formatted at 2026-06-19 18:41:04 on dist-test-slave-09fp"
server_key: "68f6551f47b099d2c17ffda13e1dd7bb"
server_key_iv: "b436e67558626ab9d288e26a5fd3ad85"
server_key_version: "encryptionkey@0"
I20260619 18:41:04.869609 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:04.888252 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:04.892846 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:04.893155 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:04.914011 6999 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37893
I20260619 18:41:04.914024 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37893 every 8 connection(s)
I20260619 18:41:04.915619 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:04.919428 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 6999
I20260619 18:41:04.919698 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:04.920082 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:04.920558 3414 external_mini_cluster.cc:1428] Setting key 42dc7f356d9ab3f8eb55d78b1437fd91
I20260619 18:41:04.935639 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b: Bootstrap starting.
Jun 19 18:41:04 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:04.939594 7066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:04.940943 7066 log.cc:824] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:04.944135 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b: No bootstrap required, opened a new log
I20260619 18:41:04.947331 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:04.922870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.3.85.190:37893)
0619 18:41:04.923940 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:04.923958 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:04.924013 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:41:04.926977 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:04.927022 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:04.927575 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:04.928002 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:04.929901 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:04.931405 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:04.933362 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:04.934171 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:04.934333 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:04.939174 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:04.939183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:04.939203 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:04.939247 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:04.942647 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:04.943460 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:04.943467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:04.943473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:04.943659 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:04.944070 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:04.944079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:04.944081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:04.944710 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:04.945043 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:04.945523 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 18:41:04.946076 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":581,"threads_started":1}
I20260619 18:41:04.953277 7066 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } }
I20260619 18:41:04.953734 7066 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:04.953850 7066 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e71c7d2f3b4fa59bd650055abe280b, State: Initialized, Role: FOLLOWER
I20260619 18:41:04.954492 7066 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } }
I20260619 18:41:04.954763 7066 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:04.954912 7066 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:04.955123 7066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:04.957587 7066 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } }
I20260619 18:41:04.958110 7066 leader_election.cc:302] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1e71c7d2f3b4fa59bd650055abe280b; no voters:
I20260619 18:41:04.958705 7066 leader_election.cc:288] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:04.958870 7071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:04.960299 7071 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [term 1 LEADER]: Becoming Leader. State: Replica: b1e71c7d2f3b4fa59bd650055abe280b, State: Running, Role: LEADER
I20260619 18:41:04.960947 7071 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } }
I20260619 18:41:04.961213 7066 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:04.963658 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1e71c7d2f3b4fa59bd650055abe280b. Latest consensus state: current_term: 1 leader_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } } }
I20260619 18:41:04.964085 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:04.964659 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e71c7d2f3b4fa59bd650055abe280b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37893 } } }
I20260619 18:41:04.964959 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:04.966709 7078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:04.972559 7078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:04.983688 7078 catalog_manager.cc:1347] Generated new cluster ID: 6d8901e0f6744d6086aef65386c8eea9
I20260619 18:41:04.983822 7078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:05.003950 7078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:05.005396 7078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:05.025820 7078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1e71c7d2f3b4fa59bd650055abe280b: Generated new TSK 0
I20260619 18:41:05.026888 7078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:41:05.078272 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:05.451095 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:05.451397 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:05.451531 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:05.459213 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:05.459388 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:05.459573 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:05.470132 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:05.471525 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:05.472992 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:05.488166 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:05.488121 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:05.488215 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:05.489683 7092 server_base.cc:1034] running on GCE node
I20260619 18:41:05.490402 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:05.491576 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:05.492787 7092 hybrid_clock.cc:648] HybridClock initialized: now 1781894465492742 us; error 38 us; skew 500 ppm
I20260619 18:41:05.493145 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:05 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894465, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:05.496585 7092 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:41:05.498275 7092 webserver.cc:460] Webserver started at http://127.3.85.129:41389/ using document root <none> and password file <none>
I20260619 18:41:05.499099 7092 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:05.499255 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:05.499585 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:05.502884 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "b5bc8fbcd1b0441da96ba15b882e342f"
format_stamp: "Formatted at 2026-06-19 18:41:05 on dist-test-slave-09fp"
server_key: "ba2d803021033a9e31c65708d6cf6d56"
server_key_iv: "0a5eabdf2bd9a60062892037d1332db8"
server_key_version: "encryptionkey@0"
I20260619 18:41:05.503778 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "b5bc8fbcd1b0441da96ba15b882e342f"
format_stamp: "Formatted at 2026-06-19 18:41:05 on dist-test-slave-09fp"
server_key: "ba2d803021033a9e31c65708d6cf6d56"
server_key_iv: "0a5eabdf2bd9a60062892037d1332db8"
server_key_version: "encryptionkey@0"
I20260619 18:41:05.509639 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:05.513867 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:05.515691 7092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:05.515920 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "b5bc8fbcd1b0441da96ba15b882e342f"
format_stamp: "Formatted at 2026-06-19 18:41:05 on dist-test-slave-09fp"
server_key: "ba2d803021033a9e31c65708d6cf6d56"
server_key_iv: "0a5eabdf2bd9a60062892037d1332db8"
server_key_version: "encryptionkey@0"
I20260619 18:41:05.516175 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:05.536449 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:05.540849 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:05.541174 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:05.542197 7092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:05.543715 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:05.543833 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:05.543988 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:05.544059 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:05.579223 7092 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:37401
I20260619 18:41:05.579260 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:37401 every 8 connection(s)
I20260619 18:41:05.580607 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:05.582788 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7092
I20260619 18:41:05.583032 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:05.583731 3414 external_mini_cluster.cc:1428] Setting key 9007aa1a0b2910b41bec7d22fce5477c
Jun 19 18:41:05 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894465, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:05.603621 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:05.583636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:51043 (local address 127.3.85.190:37893)
0619 18:41:05.584607 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:05.584612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:05.585560 (+ 948us) server_negotiation.cc:407] Connection header received
0619 18:41:05.587585 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:05.587590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:05.587681 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:05.587817 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:05.589455 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:05.590193 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:05.592633 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:05.593050 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:05.593207 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:05.597023 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:05.597028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:05.597032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:05.597067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:05.600103 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:05.600965 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:05.600971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:05.600974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:05.601105 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:05.601671 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:05.601683 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:05.601686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:05.601959 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:05.602124 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:05.603003 (+ 879us) server_negotiation.cc:299] Negotiation successful
0619 18:41:05.603246 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":717,"threads_started":1}
I20260619 18:41:05.604905 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:05.584426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37893 (local address 127.3.85.129:51043)
0619 18:41:05.585376 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:05.585432 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:05.587270 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:05.588087 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:05.588096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:05.588649 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:05.589254 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:05.589273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:05.590385 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:05.590389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:05.592448 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:05.592464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:05.593318 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:05.593328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:05.593594 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:05.594329 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:05.594359 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:05.596730 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:05.600290 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:05.600301 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:05.600322 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:05.600808 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:05.601225 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:05.601231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:05.601235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:05.601504 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:05.602102 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:05.602111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:05.602594 (+ 483us) client_negotiation.cc:769] Sending connection context
0619 18:41:05.602889 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 18:41:05.603502 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":412,"threads_started":1}
I20260619 18:41:05.606874 7220 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37893
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:05.607231 7220 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:05.608006 7220 heartbeater.cc:511] Master 127.3.85.190:37893 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:41:05.610746 7031 ts_manager.cc:194] Registered new tserver with Master: b5bc8fbcd1b0441da96ba15b882e342f (127.3.85.129:37401)
I20260619 18:41:05.612351 7031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:51043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:41:05.633807 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:05.975387 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:05.975700 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:05.975795 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:05.983294 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:05.983422 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:05.983541 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:05.993752 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:05.995080 7226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:05.996376 7226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:06.010171 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:06.010106 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:06.010192 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:06.011274 7226 server_base.cc:1034] running on GCE node
I20260619 18:41:06.011929 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:06.012913 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:06.014163 7226 hybrid_clock.cc:648] HybridClock initialized: now 1781894466014094 us; error 65 us; skew 500 ppm
I20260619 18:41:06.014585 7226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:06.018013 7226 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:41:06.019783 7226 webserver.cc:460] Webserver started at http://127.3.85.130:39075/ using document root <none> and password file <none>
I20260619 18:41:06.020671 7226 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:06.020826 7226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:06.021158 7226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:06.024425 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "29385f66dcfc4f64b6fda40a858db8eb"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "4a5f87b7a562bba2e6b8c879327990b6"
server_key_iv: "2546668ab3f36f0e0d219f4c0293872c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.025305 7226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "29385f66dcfc4f64b6fda40a858db8eb"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "4a5f87b7a562bba2e6b8c879327990b6"
server_key_iv: "2546668ab3f36f0e0d219f4c0293872c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.031116 7226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 18:41:06.035089 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.036556 7226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:06.036754 7226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "29385f66dcfc4f64b6fda40a858db8eb"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "4a5f87b7a562bba2e6b8c879327990b6"
server_key_iv: "2546668ab3f36f0e0d219f4c0293872c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.036985 7226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:06.071381 7226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:06.076134 7226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:06.076469 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:06.077520 7226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:06.079041 7226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:06.079161 7226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.079322 7226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:06.079396 7226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.114596 7226 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:38025
I20260619 18:41:06.114640 7353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:38025 every 8 connection(s)
I20260619 18:41:06.116361 7226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:06.125573 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7226
I20260619 18:41:06.125800 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:06.126529 3414 external_mini_cluster.cc:1428] Setting key 6075ad9d8f489188cc92e2531853ba9c
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:06.140338 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.119967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:36017 (local address 127.3.85.190:37893)
0619 18:41:06.120525 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.120532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.121656 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 18:41:06.123654 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.123658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.123731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.123834 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.125549 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.126877 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.129000 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.129716 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.129859 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.133581 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.133590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.133598 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.133648 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.136569 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.137722 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.137727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.137729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.137863 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.138412 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.138415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.138417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.138775 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.138889 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.139880 (+ 991us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.140068 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260619 18:41:06.142267 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.120636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37893 (local address 127.3.85.130:36017)
0619 18:41:06.121411 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:06.121460 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:06.123341 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:06.124077 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:06.124087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:06.124669 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:06.125316 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:06.125335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.127048 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.127052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:06.128844 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:06.128858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.129947 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.129952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:06.130180 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.130960 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:06.130988 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:06.133286 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:06.136779 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:06.136794 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:06.136819 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:06.137582 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:06.137985 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:06.137994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:06.138001 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:06.138290 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:06.138918 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:06.138930 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:06.139456 (+ 526us) client_negotiation.cc:769] Sending connection context
0619 18:41:06.139823 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 18:41:06.140479 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":326,"threads_started":1}
I20260619 18:41:06.143997 7354 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37893
I20260619 18:41:06.144356 7354 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:06.145130 7354 heartbeater.cc:511] Master 127.3.85.190:37893 requested a full tablet report, sending...
I20260619 18:41:06.146750 7031 ts_manager.cc:194] Registered new tserver with Master: 29385f66dcfc4f64b6fda40a858db8eb (127.3.85.130:38025)
I20260619 18:41:06.147661 7031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:36017
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:41:06.175043 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:06.521054 7360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:06.521433 7360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:06.521621 7360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:06.529549 7360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:06.529742 7360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:06.529947 7360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:06.540361 7360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:06.541888 7360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:06.543579 7360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:06.557649 7368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:06.557713 7365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:06.557724 7366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:06.563494 7360 server_base.cc:1034] running on GCE node
I20260619 18:41:06.564309 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:06.565546 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:06.566830 7360 hybrid_clock.cc:648] HybridClock initialized: now 1781894466566769 us; error 49 us; skew 500 ppm
I20260619 18:41:06.567245 7360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:06.570739 7360 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:41:06.572355 7360 webserver.cc:460] Webserver started at http://127.3.85.131:46353/ using document root <none> and password file <none>
I20260619 18:41:06.573166 7360 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:06.573304 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:06.573669 7360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:06.576860 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "b19d5cb948114a8aae0f8215aa410e42"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "b32f0eea17b707be38b2d4d3ac1a8abb"
server_key_iv: "24123aae7024ed1be71ba8bb4cc4744c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.577732 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "b19d5cb948114a8aae0f8215aa410e42"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "b32f0eea17b707be38b2d4d3ac1a8abb"
server_key_iv: "24123aae7024ed1be71ba8bb4cc4744c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.583532 7360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:41:06.587458 7375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.588973 7360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:41:06.589178 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "b19d5cb948114a8aae0f8215aa410e42"
format_stamp: "Formatted at 2026-06-19 18:41:06 on dist-test-slave-09fp"
server_key: "b32f0eea17b707be38b2d4d3ac1a8abb"
server_key_iv: "24123aae7024ed1be71ba8bb4cc4744c"
server_key_version: "encryptionkey@0"
I20260619 18:41:06.589425 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:06.612821 7360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:06.616111 7220 heartbeater.cc:503] Master 127.3.85.190:37893 was elected leader, sending a full tablet report...
I20260619 18:41:06.619088 7360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:06.619410 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:06.620561 7360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:06.622032 7360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:06.622112 7360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.622242 7360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:06.622291 7360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:06.656656 7360 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:37707
I20260619 18:41:06.656715 7487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:37707 every 8 connection(s)
I20260619 18:41:06.658034 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:06.666792 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7360
I20260619 18:41:06.667049 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:06.667505 3414 external_mini_cluster.cc:1428] Setting key 990524c03d9d2d941298fef98630a091
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:06.684070 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.661352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37893 (local address 127.3.85.131:52765)
0619 18:41:06.662370 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:06.662428 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:06.664386 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:06.665284 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:06.665294 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:06.665881 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:06.666711 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:06.666751 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.668207 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.668211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:06.670176 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:06.670190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.671322 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.671329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:06.671546 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.672228 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:06.672252 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:06.674869 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:06.678850 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:06.678862 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:06.678883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:06.679538 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:06.680175 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:06.680179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:06.680184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:06.680442 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:06.681176 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:06.681186 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:06.681694 (+ 508us) client_negotiation.cc:769] Sending connection context
0619 18:41:06.682000 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 18:41:06.682626 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":523,"threads_started":1}
I20260619 18:41:06.685284 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.660774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:52765 (local address 127.3.85.190:37893)
0619 18:41:06.661215 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.661220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.662615 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 18:41:06.664740 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.664745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.664852 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.664964 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.667035 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.667955 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.670365 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.671070 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.671173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.675159 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.675164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.675168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.675204 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.678664 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.679766 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.679775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.679779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.680019 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.680566 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.680571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.680574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.681014 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.681205 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.684676 (+ 3471us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.684924 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":205,"threads_started":1}
I20260619 18:41:06.686853 7488 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37893
I20260619 18:41:06.687240 7488 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:06.688182 7488 heartbeater.cc:511] Master 127.3.85.190:37893 requested a full tablet report, sending...
I20260619 18:41:06.689967 7031 ts_manager.cc:194] Registered new tserver with Master: b19d5cb948114a8aae0f8215aa410e42 (127.3.85.131:37707)
I20260619 18:41:06.690918 7031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:52765
I20260619 18:41:06.696682 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:06.749971 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.733959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.3.85.129:37401)
0619 18:41:06.734708 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.734723 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.734748 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:06.734806 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.734814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.735035 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.735212 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.736652 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.737671 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.739014 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.739589 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.739739 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.743875 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.743890 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.743909 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.743947 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.746992 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.747750 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.747759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.747761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.747894 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.748303 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.748308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.748311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.748667 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.748990 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.749320 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.749608 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:06.768959 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.755065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.3.85.190:37893)
0619 18:41:06.755301 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.755310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.755528 (+ 218us) server_negotiation.cc:407] Connection header received
0619 18:41:06.755730 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.755742 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.755807 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.755914 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.757214 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.757994 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.759291 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.759697 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.759836 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.763845 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.763851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.763854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.763892 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.766251 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.766956 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.766963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.766967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.767093 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.767479 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.767484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.767489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.767859 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.768108 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.768407 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.768648 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 18:41:06.769487 7031 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50122
I20260619 18:41:06.770535 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7092
I20260619 18:41:06.783510 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.771378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.3.85.129:37401)
0619 18:41:06.771618 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.771624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.771805 (+ 181us) server_negotiation.cc:407] Connection header received
0619 18:41:06.771958 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.771962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.772027 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.772120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.773302 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.774116 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.775421 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.775825 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.775928 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.778232 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.778237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.778245 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.778278 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.780841 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.781583 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.781587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.781590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.781732 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.782077 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.782080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.782082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.782322 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.782587 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.782931 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.783129 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 18:41:06.784317 7215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:06.869127 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7092
I20260619 18:41:06.886096 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7226
Jun 19 18:41:06 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:41:06.903875 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:06.887176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.3.85.130:38025)
0619 18:41:06.887690 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:06.887697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:06.887722 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:06.887789 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:06.887795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:06.888008 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:06.888142 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:06.889430 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.890538 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.891880 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:06.892461 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:06.892565 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:06.897736 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:06.897752 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:06.897769 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:06.897811 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:06.900806 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.901585 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.901592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.901594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.901725 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:06.902131 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:06.902135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:06.902137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:06.902606 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:06.902855 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:06.903219 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 18:41:06.903474 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260619 18:41:06.905293 7349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:06.989240 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7226
I20260619 18:41:07.005421 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7360
Jun 19 18:41:07 dist-test-slave-09fp krb5kdc[6986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:41:07.023182 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:07.006666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.3.85.131:37707)
0619 18:41:07.007243 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:07.007250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:07.007282 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:41:07.007347 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:07.007354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:07.007574 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:07.007717 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:07.009000 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:07.010050 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:07.011650 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:07.012274 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:07.012402 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:07.017059 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:07.017075 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:07.017091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:07.017132 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:07.020043 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:07.020894 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:07.020903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:07.020905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:07.021043 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:07.021446 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:07.021450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:07.021456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:07.021852 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:07.022113 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:07.022570 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 18:41:07.022819 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":271,"threads_started":1}
I20260619 18:41:07.024458 7483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:07.108270 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7360
I20260619 18:41:07.124291 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 6999
I20260619 18:41:07.125494 7061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:07.227622 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 6999
2026-06-19T18:41:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4082 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:07 dist-test-slave-09fp krb5kdc[7508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:07 dist-test-slave-09fp krb5kdc[7508](info): set up 2 sockets
Jun 19 18:41:07 dist-test-slave-09fp krb5kdc[7508](info): commencing operation
krb5kdc: starting...
W20260619 18:41:08.323307 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:08 dist-test-slave-09fp krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:08.534786 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35017
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:35017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:08.877213 7521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:08.877609 7521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:08.877761 7521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:08.885447 7521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:08.885618 7521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:08.885722 7521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:08.885838 7521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:08.885919 7521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:08.896525 7521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:35017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:08.897974 7521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:08.899407 7521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:08.912585 7526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:08.912556 7527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:08.912534 7529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:08.913760 7521 server_base.cc:1034] running on GCE node
I20260619 18:41:08.914675 7521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:08.916215 7521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:08.917469 7521 hybrid_clock.cc:648] HybridClock initialized: now 1781894468917405 us; error 66 us; skew 500 ppm
I20260619 18:41:08.917920 7521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:08 dist-test-slave-09fp krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:08.921182 7521 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:08.922492 7521 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:08.922646 7521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:08.922993 7521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:08.926816 7521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "d3c005ce12b747d4a9c422b11b9f3245"
format_stamp: "Formatted at 2026-06-19 18:41:08 on dist-test-slave-09fp"
server_key: "32572c048a944120deae6703844963f1"
server_key_iv: "02194370094f37a0c96a0962dfac8114"
server_key_version: "encryptionkey@0"
I20260619 18:41:08.927731 7521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "d3c005ce12b747d4a9c422b11b9f3245"
format_stamp: "Formatted at 2026-06-19 18:41:08 on dist-test-slave-09fp"
server_key: "32572c048a944120deae6703844963f1"
server_key_iv: "02194370094f37a0c96a0962dfac8114"
server_key_version: "encryptionkey@0"
I20260619 18:41:08.933332 7521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:08.937479 7535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:08.939394 7521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:08.939616 7521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "d3c005ce12b747d4a9c422b11b9f3245"
format_stamp: "Formatted at 2026-06-19 18:41:08 on dist-test-slave-09fp"
server_key: "32572c048a944120deae6703844963f1"
server_key_iv: "02194370094f37a0c96a0962dfac8114"
server_key_version: "encryptionkey@0"
I20260619 18:41:08.939868 7521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:08.973342 7521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:08.978647 7521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:08.978957 7521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:08.997730 7521 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:35017
I20260619 18:41:08.997754 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:35017 every 8 connection(s)
I20260619 18:41:08.998772 7521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:09.002908 7587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:09.005934 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7521
I20260619 18:41:09.006179 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:09.006875 3414 external_mini_cluster.cc:1428] Setting key 187d062ea0be6b0af4844d29ae6349db
I20260619 18:41:09.016140 7587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: Bootstrap starting.
I20260619 18:41:09.020452 7587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:09.021929 7587 log.cc:824] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:41:09 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:09.025731 7587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: No bootstrap required, opened a new log
I20260619 18:41:09.033834 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:09.009073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.3.85.190:35017)
0619 18:41:09.009948 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:09.009961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:09.010006 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:09.011735 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:09.011762 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:09.012253 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:09.012631 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:09.014655 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.016074 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.017944 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.018937 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.019099 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:09.024350 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:09.024363 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:09.024386 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:09.024447 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:09.028666 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:09.029479 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:09.029489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:09.029502 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:09.029739 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:09.030206 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:09.030221 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:09.030229 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:09.031046 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:09.031375 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:09.031776 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 18:41:09.032267 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":415,"threads_started":1}
I20260619 18:41:09.036638 7587 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } }
I20260619 18:41:09.037101 7587 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:09.037235 7587 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c005ce12b747d4a9c422b11b9f3245, State: Initialized, Role: FOLLOWER
I20260619 18:41:09.037793 7587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } }
I20260619 18:41:09.037999 7587 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:09.038094 7587 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:09.038221 7587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:09.040462 7587 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } }
I20260619 18:41:09.041028 7587 leader_election.cc:302] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c005ce12b747d4a9c422b11b9f3245; no voters:
I20260619 18:41:09.041762 7587 leader_election.cc:288] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:09.041896 7592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:09.042635 7592 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [term 1 LEADER]: Becoming Leader. State: Replica: d3c005ce12b747d4a9c422b11b9f3245, State: Running, Role: LEADER
I20260619 18:41:09.043300 7592 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } }
I20260619 18:41:09.044418 7587 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:09.047103 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c005ce12b747d4a9c422b11b9f3245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } } }
I20260619 18:41:09.047572 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:09.048439 7594 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c005ce12b747d4a9c422b11b9f3245. Latest consensus state: current_term: 1 leader_uuid: "d3c005ce12b747d4a9c422b11b9f3245" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c005ce12b747d4a9c422b11b9f3245" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35017 } } }
I20260619 18:41:09.048853 7594 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:09.050093 7599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:09.055806 7599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:09.065915 7599 catalog_manager.cc:1347] Generated new cluster ID: 30b77804c16f4660be3d5070974c4f0a
I20260619 18:41:09.066061 7599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:09.104991 7599 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:09.107040 7599 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:09.117431 7599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: Generated new TSK 0
I20260619 18:41:09.118607 7599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:41:09.182878 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:09.536619 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:09.537106 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:09.537294 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:09.547804 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:09.547961 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:09.548149 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:09.558586 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:09.559965 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:09.561367 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:09.574970 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:09.574993 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:09.574931 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:09.575852 7613 server_base.cc:1034] running on GCE node
I20260619 18:41:09.576499 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:09.577486 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:09.578725 7613 hybrid_clock.cc:648] HybridClock initialized: now 1781894469578669 us; error 73 us; skew 500 ppm
I20260619 18:41:09.579071 7613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:09 dist-test-slave-09fp krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:09.582674 7613 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:41:09.583953 7613 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:09.584110 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:09.584481 7613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:09.587744 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "b7446aadd5904e4f9b4660547648c557"
format_stamp: "Formatted at 2026-06-19 18:41:09 on dist-test-slave-09fp"
server_key: "9acdec0d859f06efa6dba8c10122eb10"
server_key_iv: "7b4b1304d899e44ea8f108eac85687fa"
server_key_version: "encryptionkey@0"
I20260619 18:41:09.588701 7613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "b7446aadd5904e4f9b4660547648c557"
format_stamp: "Formatted at 2026-06-19 18:41:09 on dist-test-slave-09fp"
server_key: "9acdec0d859f06efa6dba8c10122eb10"
server_key_iv: "7b4b1304d899e44ea8f108eac85687fa"
server_key_version: "encryptionkey@0"
I20260619 18:41:09.594513 7613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 18:41:09.598348 7627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:09.600036 7613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:09.600271 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "b7446aadd5904e4f9b4660547648c557"
format_stamp: "Formatted at 2026-06-19 18:41:09 on dist-test-slave-09fp"
server_key: "9acdec0d859f06efa6dba8c10122eb10"
server_key_iv: "7b4b1304d899e44ea8f108eac85687fa"
server_key_version: "encryptionkey@0"
I20260619 18:41:09.600569 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:09.622426 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:09.627744 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:09.628086 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:09.629463 7613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:09.631248 7613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:09.631395 7613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:09.631529 7613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:09.631644 7613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:09.668731 7613 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:40995
I20260619 18:41:09.668776 7739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:40995 every 8 connection(s)
I20260619 18:41:09.669749 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:09.677119 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7613
I20260619 18:41:09.677364 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:09.678158 3414 external_mini_cluster.cc:1428] Setting key b0e7c627afb52cc58cf182eb2b08c13a
Jun 19 18:41:09 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:09.695536 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:09.672532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:48797 (local address 127.3.85.190:35017)
0619 18:41:09.673031 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:09.673038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:09.674149 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 18:41:09.676118 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:09.676124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:09.676225 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:09.676355 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:09.678795 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.679707 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.682323 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.682829 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.682946 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:09.687706 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:09.687716 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:09.687724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:09.687779 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:09.691440 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:09.692367 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:09.692376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:09.692383 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:09.692535 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:09.692995 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:09.693003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:09.693010 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:09.693431 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:09.693584 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:09.694731 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0619 18:41:09.695027 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260619 18:41:09.696338 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:09.673182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35017 (local address 127.3.85.129:48797)
0619 18:41:09.673973 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:09.674014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:09.675798 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:09.676641 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:09.676655 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:09.677547 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:09.678528 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:09.678561 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.679913 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.679919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:09.682089 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:09.682115 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:09.683102 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:09.683111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:09.683462 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:09.684321 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:09.684373 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:09.687390 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:09.691677 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:09.691687 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:09.691703 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:09.692201 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:09.692668 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:09.692671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:09.692674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:09.692864 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:09.693626 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:09.693635 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:09.694029 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 18:41:09.694582 (+ 553us) client_negotiation.cc:240] Negotiation successful
0619 18:41:09.695051 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260619 18:41:09.698802 7740 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35017
I20260619 18:41:09.699271 7740 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:09.700215 7740 heartbeater.cc:511] Master 127.3.85.190:35017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:09.702948 7552 ts_manager.cc:194] Registered new tserver with Master: b7446aadd5904e4f9b4660547648c557 (127.3.85.129:40995)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:41:09.704952 7552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:48797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:41:09.727998 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:10.073642 7746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:10.074020 7746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:10.074208 7746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:10.082108 7746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:10.082294 7746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:10.082566 7746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:10.092988 7746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:10.094595 7746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:10.096134 7746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:10.110212 7754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:10.110391 7751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:10.110622 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:10.111788 7746 server_base.cc:1034] running on GCE node
I20260619 18:41:10.112498 7746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:10.113600 7746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:10.114826 7746 hybrid_clock.cc:648] HybridClock initialized: now 1781894470114753 us; error 64 us; skew 500 ppm
I20260619 18:41:10.115195 7746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:10 dist-test-slave-09fp krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:10.122167 7746 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:41:10.123581 7746 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:10.123688 7746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:10.123965 7746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:10.127249 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "56e67437e7c552ffde6d86c00e59b1bc"
server_key_iv: "ae63512d26e078f58d444efa711012cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.128122 7746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "56e67437e7c552ffde6d86c00e59b1bc"
server_key_iv: "ae63512d26e078f58d444efa711012cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.133605 7746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:41:10.137450 7760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.139178 7746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:10.139370 7746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "56e67437e7c552ffde6d86c00e59b1bc"
server_key_iv: "ae63512d26e078f58d444efa711012cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.139628 7746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:10.159382 7746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:10.163650 7746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:10.163946 7746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:10.164992 7746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:10.166501 7746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:10.166594 7746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.166736 7746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:10.166824 7746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.201683 7746 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:39309
I20260619 18:41:10.201785 7872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:39309 every 8 connection(s)
I20260619 18:41:10.202714 7746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:10.209973 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7746
I20260619 18:41:10.210228 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:10.210774 3414 external_mini_cluster.cc:1428] Setting key 7ccc5e1dcdef78d5f447acea24739b96
Jun 19 18:41:10 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:10.224706 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.205592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:60801 (local address 127.3.85.190:35017)
0619 18:41:10.206120 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.206125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.207290 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 18:41:10.209157 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.209163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.209264 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.209387 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:10.211288 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.212138 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.213998 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.214511 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.214674 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.218359 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:10.218364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:10.218367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:10.218402 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:10.221213 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.222120 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.222126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.222130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.222275 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.222828 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.222833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.222836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.223125 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:10.223287 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.224051 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.224310 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260619 18:41:10.225761 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.206215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35017 (local address 127.3.85.130:60801)
0619 18:41:10.207081 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.207132 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.208832 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.209646 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.209655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.210295 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:10.211072 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.211100 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.212326 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.212330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.213824 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.213839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.214771 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.214780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.215022 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.215782 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:10.215807 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:10.218064 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:10.221433 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:10.221444 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:10.221464 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:10.221956 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:10.222401 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:10.222407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:10.222411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:10.222701 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:10.223262 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:10.223271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:10.223650 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.223935 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.224552 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":425,"threads_started":1}
I20260619 18:41:10.227603 7873 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35017
I20260619 18:41:10.227970 7873 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:10.228703 7873 heartbeater.cc:511] Master 127.3.85.190:35017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:41:10.230533 7552 ts_manager.cc:194] Registered new tserver with Master: 15f4ae2e16b548c5a6a8fc5b29e89bd6 (127.3.85.130:39309)
I20260619 18:41:10.231370 7552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:60801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:41:10.257351 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:10.612987 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:10.613296 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:10.613394 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:10.621327 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:10.621454 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:10.621572 7879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:10.631717 7879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:10.633044 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:10.634321 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:10.648233 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:10.648164 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:10.648257 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:10.648996 7879 server_base.cc:1034] running on GCE node
I20260619 18:41:10.649806 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:10.650959 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:10.652210 7879 hybrid_clock.cc:648] HybridClock initialized: now 1781894470652143 us; error 45 us; skew 500 ppm
I20260619 18:41:10.652597 7879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:10 dist-test-slave-09fp krb5kdc[7508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:10.656100 7879 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:41:10.657325 7879 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:10.657480 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:10.657815 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:10.660916 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "0db760b97d684dd49b7136c73604a0b7"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "aa04301a78455d03c5c830967b0fe7a7"
server_key_iv: "59b64c07cfd19ff4a3702eafaf55f1f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.661798 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "0db760b97d684dd49b7136c73604a0b7"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "aa04301a78455d03c5c830967b0fe7a7"
server_key_iv: "59b64c07cfd19ff4a3702eafaf55f1f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.667379 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 18:41:10.671470 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.673089 7879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:10.673305 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "0db760b97d684dd49b7136c73604a0b7"
format_stamp: "Formatted at 2026-06-19 18:41:10 on dist-test-slave-09fp"
server_key: "aa04301a78455d03c5c830967b0fe7a7"
server_key_iv: "59b64c07cfd19ff4a3702eafaf55f1f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:10.673557 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:10.688419 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:10.692842 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:10.693148 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:10.694351 7879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:10.695885 7879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:10.696040 7879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.696171 7879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:10.696267 7879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:10.708513 7740 heartbeater.cc:503] Master 127.3.85.190:35017 was elected leader, sending a full tablet report...
I20260619 18:41:10.738678 7879 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:33011
I20260619 18:41:10.738730 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:33011 every 8 connection(s)
I20260619 18:41:10.739729 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:10.748044 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 7879
I20260619 18:41:10.748265 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:10.748731 3414 external_mini_cluster.cc:1428] Setting key 802e1a30526f7729efe21abc5125cd8d
Jun 19 18:41:10 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:10.762769 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.742399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:59297 (local address 127.3.85.190:35017)
0619 18:41:10.742878 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.742883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.744013 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 18:41:10.746168 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.746172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.746250 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.746353 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:10.748289 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.749157 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.751691 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.752397 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.752502 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.756288 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:10.756293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:10.756297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:10.756335 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:10.758939 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.759992 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.759998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.760002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.760231 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.760835 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.760841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.760843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.761199 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:10.761340 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.762114 (+ 774us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.762312 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":226,"threads_started":1}
I20260619 18:41:10.763847 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.742959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35017 (local address 127.3.85.131:59297)
0619 18:41:10.743817 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.743892 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.745838 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.746647 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.746656 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.747243 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:10.748014 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.748046 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.749391 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.749395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.751506 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.751522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.752669 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.752676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.752889 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.753559 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:10.753582 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:10.755985 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:10.759132 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:10.759142 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:10.759160 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:10.759702 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:10.760383 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:10.760386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:10.760389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:10.760610 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:10.761338 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:10.761345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:10.761739 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.761990 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.762405 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":315,"threads_started":1}
I20260619 18:41:10.765678 8006 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35017
I20260619 18:41:10.766239 8006 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:10.767282 8006 heartbeater.cc:511] Master 127.3.85.190:35017 requested a full tablet report, sending...
I20260619 18:41:10.768999 7551 ts_manager.cc:194] Registered new tserver with Master: 0db760b97d684dd49b7136c73604a0b7 (127.3.85.131:33011)
I20260619 18:41:10.770005 7551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:59297
I20260619 18:41:10.777940 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:10.796352 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.782918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.3.85.190:35017)
0619 18:41:10.783090 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.783095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.783741 (+ 646us) server_negotiation.cc:407] Connection header received
0619 18:41:10.783859 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.783862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.783932 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.784030 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:10.785596 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.786783 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.788122 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.788532 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.788634 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.791079 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:10.791084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:10.791088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:10.791126 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:10.793732 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.794499 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.794507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.794515 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.794635 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:10.795023 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:10.795026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:10.795028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:10.795340 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:10.795509 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.795869 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.796132 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:41:10.801304 7551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:10.808537 7551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:10.842639 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.822143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36872)
0619 18:41:10.829497 (+ 7354us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.829554 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.829779 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.831774 (+ 1995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.831781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.831813 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:10.832276 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.832290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.835685 (+ 3395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.835696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.839250 (+ 3554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.839280 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.840966 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.840973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.841206 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.841264 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.841914 (+ 650us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.842149 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7057,"mutex_wait_us":204,"thread_start_us":6879,"threads_started":1}
I20260619 18:41:10.842967 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.822376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.3.85.129:40995)
0619 18:41:10.831025 (+ 8649us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.831034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.831074 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:41:10.831197 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.831204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.831431 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.831771 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:10.832543 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.835444 (+ 2901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.839598 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.840731 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.840907 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.841163 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.842298 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.842525 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8404,"thread_start_us":8238,"threads_started":1}
I20260619 18:41:10.843255 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.822499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39309 (local address 127.0.0.1:53590)
0619 18:41:10.830849 (+ 8350us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.830871 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.831008 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.832268 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.832274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.832304 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:10.832946 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.832965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.835915 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.835925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.839749 (+ 3824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.839764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.842123 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.842130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.842341 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.842380 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.842575 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.842744 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8219,"mutex_wait_us":105,"thread_start_us":8064,"threads_started":1}
I20260619 18:41:10.844139 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.823143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.3.85.130:39309)
0619 18:41:10.830416 (+ 7273us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.830428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.831085 (+ 657us) server_negotiation.cc:407] Connection header received
0619 18:41:10.831422 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.831436 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.831752 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.831999 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:10.833247 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.835621 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.839952 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.841551 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.841704 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.841930 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.843505 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.843700 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7117,"thread_start_us":6988,"threads_started":1}
I20260619 18:41:10.847599 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.832281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.3.85.131:33011)
0619 18:41:10.837403 (+ 5122us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.837415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.838188 (+ 773us) server_negotiation.cc:407] Connection header received
0619 18:41:10.838308 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.838317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.838672 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.838890 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:10.839724 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.841447 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.844750 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.846363 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.846608 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.846867 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.847020 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.847204 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4915,"thread_start_us":4748,"threads_started":1}
I20260619 18:41:10.848062 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.832210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:33011 (local address 127.0.0.1:60778)
0619 18:41:10.837860 (+ 5650us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.837883 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.838048 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.838878 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.838884 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.838916 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:10.839485 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.839507 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.841720 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.841727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.844473 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.844556 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.846648 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.846655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.846825 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.846862 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.846965 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.847123 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5458,"mutex_wait_us":3,"thread_start_us":5231,"threads_started":1}
I20260619 18:41:10.847581 7675 tablet_service.cc:1467] Processing CreateTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff (DEFAULT_TABLE table=test-table [id=0b9d55776d0e46da947d252983649ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:10.848271 7808 tablet_service.cc:1467] Processing CreateTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff (DEFAULT_TABLE table=test-table [id=0b9d55776d0e46da947d252983649ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:10.849273 7675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8ad09b77cb4e59834f3305c3bafaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:10.849956 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8ad09b77cb4e59834f3305c3bafaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:10.852823 7941 tablet_service.cc:1467] Processing CreateTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff (DEFAULT_TABLE table=test-table [id=0b9d55776d0e46da947d252983649ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:10.854935 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae8ad09b77cb4e59834f3305c3bafaff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:10.862800 8025 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Bootstrap starting.
I20260619 18:41:10.865731 8026 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Bootstrap starting.
I20260619 18:41:10.866665 8025 tablet_bootstrap.cc:654] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:10.868150 8025 log.cc:824] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:10.869660 8026 tablet_bootstrap.cc:654] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:10.871038 8026 log.cc:824] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:10.871176 8028 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Bootstrap starting.
I20260619 18:41:10.872257 8025 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: No bootstrap required, opened a new log
I20260619 18:41:10.872623 8025 ts_tablet_manager.cc:1397] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260619 18:41:10.874316 8028 tablet_bootstrap.cc:654] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:10.875768 8028 log.cc:824] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:10.877517 8026 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: No bootstrap required, opened a new log
I20260619 18:41:10.877880 8026 ts_tablet_manager.cc:1397] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260619 18:41:10.883795 8025 raft_consensus.cc:348] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.884343 8025 raft_consensus.cc:374] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:10.884488 8025 raft_consensus.cc:729] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15f4ae2e16b548c5a6a8fc5b29e89bd6, State: Initialized, Role: FOLLOWER
I20260619 18:41:10.885329 8028 tablet_bootstrap.cc:492] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: No bootstrap required, opened a new log
I20260619 18:41:10.885250 8025 consensus_queue.cc:260] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.885689 8028 ts_tablet_manager.cc:1397] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260619 18:41:10.890134 8026 raft_consensus.cc:348] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.890909 8026 raft_consensus.cc:374] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:10.891058 8026 raft_consensus.cc:729] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7446aadd5904e4f9b4660547648c557, State: Initialized, Role: FOLLOWER
I20260619 18:41:10.891289 7873 heartbeater.cc:503] Master 127.3.85.190:35017 was elected leader, sending a full tablet report...
I20260619 18:41:10.891774 8026 consensus_queue.cc:260] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.893683 8026 ts_tablet_manager.cc:1428] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260619 18:41:10.895154 8025 ts_tablet_manager.cc:1428] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260619 18:41:10.894948 8028 raft_consensus.cc:348] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.895460 8028 raft_consensus.cc:374] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:10.895614 8028 raft_consensus.cc:729] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db760b97d684dd49b7136c73604a0b7, State: Initialized, Role: FOLLOWER
I20260619 18:41:10.896427 8028 consensus_queue.cc:260] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.897670 8006 heartbeater.cc:503] Master 127.3.85.190:35017 was elected leader, sending a full tablet report...
I20260619 18:41:10.898366 8028 ts_tablet_manager.cc:1428] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260619 18:41:10.922418 7741 tablet.cc:2369] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:10.954916 7874 tablet.cc:2369] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:10.975809 8033 raft_consensus.cc:482] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:10.976090 8033 raft_consensus.cc:504] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:10.977743 8033 leader_election.cc:288] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7446aadd5904e4f9b4660547648c557 (127.3.85.129:40995), 15f4ae2e16b548c5a6a8fc5b29e89bd6 (127.3.85.130:39309)
I20260619 18:41:10.984754 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.978178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39309 (local address 127.3.85.131:49591)
0619 18:41:10.978607 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.978637 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.978796 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.979189 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.979197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.979238 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:10.979451 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.979466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.980795 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.980801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.982668 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.982683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.983828 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.983837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.984079 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.984150 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.984296 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.984437 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":76}
I20260619 18:41:10.984756 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.978429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49591 (local address 127.3.85.130:39309)
0619 18:41:10.978661 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.978681 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.978706 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:10.978962 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.978969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.979053 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.979177 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:10.979624 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.980654 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.982848 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.983651 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.983791 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.983845 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.984339 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.984484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 18:41:10.984781 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.978236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:39875 (local address 127.3.85.129:40995)
0619 18:41:10.978471 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:10.978479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:10.979165 (+ 686us) server_negotiation.cc:407] Connection header received
0619 18:41:10.979338 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:10.979344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:10.979419 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:10.979516 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:10.980166 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.981048 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.982961 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.983781 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.983975 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.984071 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:10.984434 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:41:10.984564 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:41:10.985553 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:10.978191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.3.85.131:39875)
0619 18:41:10.979036 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:10.979052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:10.979203 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:10.979583 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:10.979587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:10.979613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:10.979956 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.979968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.981194 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.981198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.982791 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:10.982808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:10.983998 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:10.984003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:10.984216 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:10.984275 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 18:41:10.985058 (+ 783us) client_negotiation.cc:240] Negotiation successful
0619 18:41:10.985227 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":85,"thread_start_us":488,"threads_started":1}
I20260619 18:41:10.985975 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8ad09b77cb4e59834f3305c3bafaff" candidate_uuid: "0db760b97d684dd49b7136c73604a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" is_pre_election: true
I20260619 18:41:10.986691 7828 raft_consensus.cc:2454] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db760b97d684dd49b7136c73604a0b7 in term 0.
I20260619 18:41:10.986647 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8ad09b77cb4e59834f3305c3bafaff" candidate_uuid: "0db760b97d684dd49b7136c73604a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7446aadd5904e4f9b4660547648c557" is_pre_election: true
I20260619 18:41:10.987250 7695 raft_consensus.cc:2454] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db760b97d684dd49b7136c73604a0b7 in term 0.
I20260619 18:41:10.987540 7894 leader_election.cc:302] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db760b97d684dd49b7136c73604a0b7, 15f4ae2e16b548c5a6a8fc5b29e89bd6; no voters:
I20260619 18:41:10.988019 8033 raft_consensus.cc:2781] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:10.988185 8033 raft_consensus.cc:482] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:10.988283 8033 raft_consensus.cc:3037] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:10.990821 8033 raft_consensus.cc:504] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
W20260619 18:41:10.991567 8007 tablet.cc:2369] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:10.991637 8033 leader_election.cc:288] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [CANDIDATE]: Term 1 election: Requested vote from peers b7446aadd5904e4f9b4660547648c557 (127.3.85.129:40995), 15f4ae2e16b548c5a6a8fc5b29e89bd6 (127.3.85.130:39309)
I20260619 18:41:10.992084 7695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8ad09b77cb4e59834f3305c3bafaff" candidate_uuid: "0db760b97d684dd49b7136c73604a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7446aadd5904e4f9b4660547648c557"
I20260619 18:41:10.992340 7695 raft_consensus.cc:3037] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:10.992611 7828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae8ad09b77cb4e59834f3305c3bafaff" candidate_uuid: "0db760b97d684dd49b7136c73604a0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6"
I20260619 18:41:10.992833 7828 raft_consensus.cc:3037] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:10.994844 7695 raft_consensus.cc:2454] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db760b97d684dd49b7136c73604a0b7 in term 1.
I20260619 18:41:10.995249 7828 raft_consensus.cc:2454] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db760b97d684dd49b7136c73604a0b7 in term 1.
I20260619 18:41:10.995396 7897 leader_election.cc:302] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db760b97d684dd49b7136c73604a0b7, b7446aadd5904e4f9b4660547648c557; no voters:
I20260619 18:41:10.995800 8033 raft_consensus.cc:2781] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:10.996284 8033 raft_consensus.cc:686] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0db760b97d684dd49b7136c73604a0b7, State: Running, Role: LEADER
I20260619 18:41:10.996886 8033 consensus_queue.cc:237] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } }
I20260619 18:41:11.001734 7552 catalog_manager.cc:5495] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db760b97d684dd49b7136c73604a0b7 (127.3.85.131). New cstate: current_term: 1 leader_uuid: "0db760b97d684dd49b7136c73604a0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db760b97d684dd49b7136c73604a0b7" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33011 } health_report { overall_health: HEALTHY } } }
I20260619 18:41:11.042598 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.036770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.3.85.131:33011)
0619 18:41:11.037056 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:11.037062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:11.037084 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:11.037299 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:11.037302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:11.037380 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:11.037547 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:11.037946 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.038700 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.040151 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.040558 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.040671 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:11.041279 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:11.041841 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:11.041959 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:11.042105 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:41:11.042183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 18:41:11.051690 7828 raft_consensus.cc:1261] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 1 FOLLOWER]: Refusing update from remote peer 0db760b97d684dd49b7136c73604a0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:11.051692 7695 raft_consensus.cc:1261] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 1 FOLLOWER]: Refusing update from remote peer 0db760b97d684dd49b7136c73604a0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:11.052884 8035 consensus_queue.cc:1035] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7446aadd5904e4f9b4660547648c557" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:11.053251 8033 consensus_queue.cc:1035] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15f4ae2e16b548c5a6a8fc5b29e89bd6" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:11.083381 7551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:11.083706 7551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:11.087792 7551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c005ce12b747d4a9c422b11b9f3245: Sending DeleteTablet for 3 replicas of tablet ae8ad09b77cb4e59834f3305c3bafaff
I20260619 18:41:11.089135 7808 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:53590
I20260619 18:41:11.089134 7675 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:36872
I20260619 18:41:11.089236 7941 tablet_service.cc:1514] Processing DeleteTablet for tablet ae8ad09b77cb4e59834f3305c3bafaff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:60778
I20260619 18:41:11.092379 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7613
I20260619 18:41:11.102274 8049 tablet_replica.cc:330] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: stopping tablet replica
I20260619 18:41:11.102638 8047 tablet_replica.cc:330] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: stopping tablet replica
I20260619 18:41:11.103338 8047 raft_consensus.cc:2229] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:11.103529 8049 raft_consensus.cc:2229] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:11.104264 8047 raft_consensus.cc:2258] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:11.105142 8048 tablet_replica.cc:330] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: stopping tablet replica
I20260619 18:41:11.105643 8048 raft_consensus.cc:2229] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:11.106289 8048 raft_consensus.cc:2258] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:11.107295 8047 ts_tablet_manager.cc:1905] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:11.108403 8049 raft_consensus.cc:2258] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:11.108647 8048 ts_tablet_manager.cc:1905] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:41:11 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:11.116194 8048 ts_tablet_manager.cc:1918] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:11.116518 8048 log.cc:1196] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/ae8ad09b77cb4e59834f3305c3bafaff
I20260619 18:41:11.117230 8048 ts_tablet_manager.cc:1939] T ae8ad09b77cb4e59834f3305c3bafaff P 15f4ae2e16b548c5a6a8fc5b29e89bd6: Deleting consensus metadata
I20260619 18:41:11.118173 8049 ts_tablet_manager.cc:1905] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:11.119377 7536 catalog_manager.cc:4841] TS 15f4ae2e16b548c5a6a8fc5b29e89bd6 (127.3.85.130:39309): tablet ae8ad09b77cb4e59834f3305c3bafaff (table test-table [id=0b9d55776d0e46da947d252983649ed6]) successfully deleted
I20260619 18:41:11.119856 8047 ts_tablet_manager.cc:1918] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:11.120102 8047 log.cc:1196] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/ae8ad09b77cb4e59834f3305c3bafaff
I20260619 18:41:11.120745 8047 ts_tablet_manager.cc:1939] T ae8ad09b77cb4e59834f3305c3bafaff P 0db760b97d684dd49b7136c73604a0b7: Deleting consensus metadata
I20260619 18:41:11.122625 7536 catalog_manager.cc:4841] TS 0db760b97d684dd49b7136c73604a0b7 (127.3.85.131:33011): tablet ae8ad09b77cb4e59834f3305c3bafaff (table test-table [id=0b9d55776d0e46da947d252983649ed6]) successfully deleted
I20260619 18:41:11.124125 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.094219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.3.85.129:40995)
0619 18:41:11.094516 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:11.094526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:11.099207 (+ 4681us) server_negotiation.cc:407] Connection header received
0619 18:41:11.099448 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:11.099457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:11.099612 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:11.099771 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:11.104113 (+ 4342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.105328 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.107317 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.108033 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.108145 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:11.114287 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:11.114308 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:11.114330 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:11.114383 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:11.119981 (+ 5598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.121055 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.121071 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.121074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.121299 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.121828 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.121834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.121837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.122338 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:11.122720 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:11.123384 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 18:41:11.123679 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:41:11.125828 7735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:11.225519 8049 ts_tablet_manager.cc:1918] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:11.225776 8049 log.cc:1196] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/ae8ad09b77cb4e59834f3305c3bafaff
I20260619 18:41:11.226300 8049 ts_tablet_manager.cc:1939] T ae8ad09b77cb4e59834f3305c3bafaff P b7446aadd5904e4f9b4660547648c557: Deleting consensus metadata
I20260619 18:41:11.226367 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7613
W20260619 18:41:11.250962 7539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:40995 (error 108)
I20260619 18:41:11.251255 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7746
I20260619 18:41:11.253412 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.252898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36896)
0619 18:41:11.253108 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.253232 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 18:41:11.253695 7539 catalog_manager.cc:4584] TS b7446aadd5904e4f9b4660547648c557 (127.3.85.129:40995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae8ad09b77cb4e59834f3305c3bafaff: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Jun 19 18:41:11 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:41:11.267572 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.252473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.3.85.130:39309)
0619 18:41:11.252638 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:11.252643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:11.252659 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:41:11.252817 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:11.252823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:11.252919 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:11.253023 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:11.254422 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.255293 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.256507 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.256897 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.257036 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:11.261479 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:11.261495 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:11.261512 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:11.261556 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:11.264600 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.265401 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.265408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.265411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.265546 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.265947 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.265953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.265956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.266357 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:11.266703 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:11.266998 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 18:41:11.267231 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:41:11.268321 7868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:11.307585 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36910)
0619 18:41:11.307263 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.307391 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:41:11.308869 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.308416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36924)
0619 18:41:11.308599 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.308720 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 18:41:11.359320 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7746
I20260619 18:41:11.370004 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.369357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36928)
0619 18:41:11.369663 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.369787 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 18:41:11.371369 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.370870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36934)
0619 18:41:11.371068 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.371171 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 18:41:11.377305 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7879
Jun 19 18:41:11 dist-test-slave-09fp krb5kdc[7508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:41:11.396232 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.378256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.3.85.131:33011)
0619 18:41:11.378560 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:11.378565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:11.378585 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:41:11.378757 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:11.378761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:11.378827 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:11.378925 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:11.380832 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.381688 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.383780 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:11.384327 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:11.384444 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:11.389765 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:11.389781 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:11.389797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:11.389844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:11.393026 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.393842 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.393851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.393853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.394022 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:11.394475 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:11.394483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:11.394489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:11.394949 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:11.395295 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:11.395591 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 18:41:11.395825 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 18:41:11.397097 8001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:11.452155 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.451650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36944)
0619 18:41:11.451874 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.451967 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 18:41:11.453281 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:11.452879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40995 (local address 127.0.0.1:36946)
0619 18:41:11.453097 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:11.453166 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:40995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 18:41:11.519065 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7879
I20260619 18:41:11.538585 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 7521
I20260619 18:41:11.539839 7582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:11.649745 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 7521
2026-06-19T18:41:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4422 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T18:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:11Z Disabled control of system clock
I20260619 18:41:11.722157 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46483
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:41:12.062731 8059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:12.063038 8059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:12.063123 8059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:12.070425 8059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:12.070586 8059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:12.070626 8059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:12.070684 8059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:12.070729 8059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:12.080082 8059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:12.081408 8059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:12.083288 8059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:12.095788 8065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:12.095764 8064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:12.095831 8067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:12.097220 8059 server_base.cc:1034] running on GCE node
I20260619 18:41:12.098184 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:12.099895 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:12.101125 8059 hybrid_clock.cc:648] HybridClock initialized: now 1781894472101082 us; error 41 us; skew 500 ppm
I20260619 18:41:12.101462 8059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:12.103948 8059 webserver.cc:460] Webserver started at http://127.3.85.190:36899/ using document root <none> and password file <none>
I20260619 18:41:12.104708 8059 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:12.104807 8059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:12.105089 8059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:12.108223 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "0fc5e026039645a78f21289a2793ce2a"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "668650d20c8458f19bd874e9c34020ac"
server_key_iv: "870ce04f9da149b1ad1d4c3748363134"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.109062 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "0fc5e026039645a78f21289a2793ce2a"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "668650d20c8458f19bd874e9c34020ac"
server_key_iv: "870ce04f9da149b1ad1d4c3748363134"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.114883 8059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:41:12.118691 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:12.120225 8059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:12.120383 8059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "0fc5e026039645a78f21289a2793ce2a"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "668650d20c8458f19bd874e9c34020ac"
server_key_iv: "870ce04f9da149b1ad1d4c3748363134"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.120594 8059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:12.126925 8059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:12.127784 8059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:12.128051 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:12.147706 8059 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46483
I20260619 18:41:12.147724 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46483 every 8 connection(s)
I20260619 18:41:12.149209 8059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:12.153327 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:12.153563 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8059
I20260619 18:41:12.153786 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:12.154515 3414 external_mini_cluster.cc:1428] Setting key 4cac7af826ae72dbb1f25ec3e96a0a86
I20260619 18:41:12.163844 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:12.156766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.3.85.190:46483)
0619 18:41:12.157952 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:12.157965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:12.158010 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:12.159746 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:12.159775 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:12.160287 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:12.160688 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:12.161960 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:12.161967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:12.161982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:12.162001 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:12.162078 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:12.162105 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:12.162308 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:12.162417 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 18:41:12.162830 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":446,"threads_started":1}
I20260619 18:41:12.167850 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: Bootstrap starting.
I20260619 18:41:12.171240 8125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:12.172335 8125 log.cc:824] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:12.175416 8125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: No bootstrap required, opened a new log
I20260619 18:41:12.183027 8125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } }
I20260619 18:41:12.183493 8125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:12.183605 8125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc5e026039645a78f21289a2793ce2a, State: Initialized, Role: FOLLOWER
I20260619 18:41:12.184422 8125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } }
I20260619 18:41:12.184732 8125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:12.184899 8125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:12.185112 8125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:12.187454 8125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } }
I20260619 18:41:12.188030 8125 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc5e026039645a78f21289a2793ce2a; no voters:
I20260619 18:41:12.188696 8125 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:12.188879 8130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:12.189654 8130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [term 1 LEADER]: Becoming Leader. State: Replica: 0fc5e026039645a78f21289a2793ce2a, State: Running, Role: LEADER
I20260619 18:41:12.190308 8130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } }
I20260619 18:41:12.191095 8125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:12.193382 8132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc5e026039645a78f21289a2793ce2a. Latest consensus state: current_term: 1 leader_uuid: "0fc5e026039645a78f21289a2793ce2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } } }
I20260619 18:41:12.193887 8132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:12.193684 8131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc5e026039645a78f21289a2793ce2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc5e026039645a78f21289a2793ce2a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46483 } } }
I20260619 18:41:12.193990 8131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:12.197141 8139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:12.201894 8139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:12.211289 8139 catalog_manager.cc:1347] Generated new cluster ID: c6ffea8fee684fa6a020cd5ac0f9c593
I20260619 18:41:12.211484 8139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:12.227685 8139 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:12.228192 8139 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:12.243177 8139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: Generated new TSK 0
I20260619 18:41:12.244148 8139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:12.257009 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--builtin_ntp_servers=127.3.85.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:41:12.603286 8149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:12.603610 8149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:12.603775 8149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:12.611701 8149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:12.611838 8149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:12.612049 8149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:12.621779 8149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:12.623418 8149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:12.625609 8149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:12.639685 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:12.639830 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:12.639771 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:12.641060 8149 server_base.cc:1034] running on GCE node
I20260619 18:41:12.641741 8149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:12.642831 8149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:12.644024 8149 hybrid_clock.cc:648] HybridClock initialized: now 1781894472643969 us; error 56 us; skew 500 ppm
I20260619 18:41:12.644404 8149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:12.647117 8149 webserver.cc:460] Webserver started at http://127.3.85.129:44379/ using document root <none> and password file <none>
I20260619 18:41:12.647882 8149 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:12.647980 8149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:12.648310 8149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:12.651503 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "d7ad276c0f484996aac2c61189d91747"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "1521a00a8f4851ccba393b665afbc196"
server_key_iv: "6b8f5b3c2734b0089d920e725d640477"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.652397 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "d7ad276c0f484996aac2c61189d91747"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "1521a00a8f4851ccba393b665afbc196"
server_key_iv: "6b8f5b3c2734b0089d920e725d640477"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.658663 8149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:41:12.662575 8163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:12.664188 8149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:12.664386 8149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "d7ad276c0f484996aac2c61189d91747"
format_stamp: "Formatted at 2026-06-19 18:41:12 on dist-test-slave-09fp"
server_key: "1521a00a8f4851ccba393b665afbc196"
server_key_iv: "6b8f5b3c2734b0089d920e725d640477"
server_key_version: "encryptionkey@0"
I20260619 18:41:12.664664 8149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:12.670637 8149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:12.671458 8149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:12.671713 8149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:12.672610 8149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:12.674163 8149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:12.674260 8149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:12.674340 8149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:12.674396 8149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:12.714175 8149 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:35657
I20260619 18:41:12.714274 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:35657 every 8 connection(s)
I20260619 18:41:12.715842 8149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:12.718587 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8149
I20260619 18:41:12.718806 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:12.719667 3414 external_mini_cluster.cc:1428] Setting key 3f0b8a20a5627be69013114c70d1ebbc
I20260619 18:41:12.722846 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--builtin_ntp_servers=127.3.85.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:41:12.725718 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:12.718854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:59633 (local address 127.3.85.190:46483)
0619 18:41:12.719423 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:12.719431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:12.721114 (+ 1683us) server_negotiation.cc:407] Connection header received
0619 18:41:12.722945 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:12.722950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:12.723020 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:12.723152 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:12.724862 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:12.724867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:12.724874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:12.724897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:12.724939 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:12.724952 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:12.725097 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:12.725321 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:41:12.725393 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
I20260619 18:41:12.726877 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:12.719688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46483 (local address 127.3.85.129:59633)
0619 18:41:12.720936 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:12.720984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:12.722639 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:12.723434 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:12.723443 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:12.723614 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:12.724468 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:12.724471 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:12.724521 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:12.724660 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:12.725061 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:12.725074 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:12.725083 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:41:12.725397 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 18:41:12.725735 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":377,"threads_started":1}
I20260619 18:41:12.728616 8276 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46483
I20260619 18:41:12.728941 8276 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:12.729544 8276 heartbeater.cc:511] Master 127.3.85.190:46483 requested a full tablet report, sending...
I20260619 18:41:12.731907 8090 ts_manager.cc:194] Registered new tserver with Master: d7ad276c0f484996aac2c61189d91747 (127.3.85.129:35657)
W20260619 18:41:13.066289 8280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:13.066634 8280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:13.066725 8280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:13.074172 8280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:13.074291 8280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:13.074405 8280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:13.084160 8280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:13.085439 8280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:13.087297 8280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:13.101367 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:13.101491 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:13.101464 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:13.102370 8280 server_base.cc:1034] running on GCE node
I20260619 18:41:13.103055 8280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:13.104018 8280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:13.105283 8280 hybrid_clock.cc:648] HybridClock initialized: now 1781894473105193 us; error 68 us; skew 500 ppm
I20260619 18:41:13.105645 8280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:13.108144 8280 webserver.cc:460] Webserver started at http://127.3.85.130:42457/ using document root <none> and password file <none>
I20260619 18:41:13.108927 8280 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:13.109028 8280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:13.109401 8280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:13.112496 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "6d2330627af04059b9744e2e1acd755b"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "7724b6c51de4a8e54a52ab7b3e370771"
server_key_iv: "3676b00381e0d399c12945071e427ca9"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.113361 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "6d2330627af04059b9744e2e1acd755b"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "7724b6c51de4a8e54a52ab7b3e370771"
server_key_iv: "3676b00381e0d399c12945071e427ca9"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.119153 8280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 18:41:13.122941 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.124564 8280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:41:13.124770 8280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "6d2330627af04059b9744e2e1acd755b"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "7724b6c51de4a8e54a52ab7b3e370771"
server_key_iv: "3676b00381e0d399c12945071e427ca9"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.125028 8280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:13.131290 8280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:13.132220 8280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:13.132530 8280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:13.133564 8280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:13.135252 8280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:13.135382 8280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.135528 8280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:13.135617 8280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.174281 8280 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:46253
I20260619 18:41:13.174305 8406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:46253 every 8 connection(s)
I20260619 18:41:13.176033 8280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:13.185876 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.179138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:38411 (local address 127.3.85.190:46483)
0619 18:41:13.179403 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.179410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.181181 (+ 1771us) server_negotiation.cc:407] Connection header received
0619 18:41:13.183238 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.183243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.183303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.183421 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.185197 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.185202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.185205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.185225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.185261 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.185274 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.185412 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.185605 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.185661 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:41:13.186220 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8280
I20260619 18:41:13.186386 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:13.186962 3414 external_mini_cluster.cc:1428] Setting key 5d0e9cef37ce82cf60788151141d2d5b
I20260619 18:41:13.187395 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.179817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46483 (local address 127.3.85.130:38411)
0619 18:41:13.180974 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.181019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.182866 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.183857 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.183867 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.183982 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.184821 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.184824 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.184868 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.184974 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.185374 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.185382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.185388 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.185656 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.185976 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":361,"threads_started":1}
I20260619 18:41:13.189008 8407 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46483
I20260619 18:41:13.189354 8407 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:13.190008 8407 heartbeater.cc:511] Master 127.3.85.190:46483 requested a full tablet report, sending...
I20260619 18:41:13.190217 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--builtin_ntp_servers=127.3.85.148:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:41:13.191977 8090 ts_manager.cc:194] Registered new tserver with Master: 6d2330627af04059b9744e2e1acd755b (127.3.85.130:46253)
W20260619 18:41:13.540961 8410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:13.541380 8410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:13.541478 8410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:13.549031 8410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:13.549162 8410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:13.549275 8410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:13.558671 8410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:13.559959 8410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:13.561798 8410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:13.576277 8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:13.576262 8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:13.576529 8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:13.577303 8410 server_base.cc:1034] running on GCE node
I20260619 18:41:13.577972 8410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:13.579118 8410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:13.580384 8410 hybrid_clock.cc:648] HybridClock initialized: now 1781894473580315 us; error 68 us; skew 500 ppm
I20260619 18:41:13.580765 8410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:13.583578 8410 webserver.cc:460] Webserver started at http://127.3.85.131:35707/ using document root <none> and password file <none>
I20260619 18:41:13.584408 8410 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:13.584558 8410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:13.584906 8410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:13.588065 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "5417f4cdc9c24e4e84f8b546efef9cfa"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "ea584139110ce68fe071e3cf23b950d9"
server_key_iv: "b2207ec6f30d3399f39db4c792e6f4c0"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.588960 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "5417f4cdc9c24e4e84f8b546efef9cfa"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "ea584139110ce68fe071e3cf23b950d9"
server_key_iv: "b2207ec6f30d3399f39db4c792e6f4c0"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.595026 8410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:41:13.599349 8424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.600919 8410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:13.601141 8410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "5417f4cdc9c24e4e84f8b546efef9cfa"
format_stamp: "Formatted at 2026-06-19 18:41:13 on dist-test-slave-09fp"
server_key: "ea584139110ce68fe071e3cf23b950d9"
server_key_iv: "b2207ec6f30d3399f39db4c792e6f4c0"
server_key_version: "encryptionkey@0"
I20260619 18:41:13.601413 8410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:13.607745 8410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:13.608664 8410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:13.608963 8410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:13.609959 8410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:13.611497 8410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:13.611627 8410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.611766 8410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:13.611855 8410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:13.648494 8410 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:44635
I20260619 18:41:13.648524 8536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:44635 every 8 connection(s)
I20260619 18:41:13.650031 8410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:13.653529 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8410
I20260619 18:41:13.653707 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:13.654137 3414 external_mini_cluster.cc:1428] Setting key c0726b133b26cca5ca5bc9e509937af3
I20260619 18:41:13.659337 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.652922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:41431 (local address 127.3.85.190:46483)
0619 18:41:13.653148 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.653155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.655049 (+ 1894us) server_negotiation.cc:407] Connection header received
0619 18:41:13.656854 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.656860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.656945 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.657079 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.658510 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.658517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.658522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.658553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.658587 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.658600 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.658734 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.658961 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.659033 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:41:13.660385 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.653563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46483 (local address 127.3.85.131:41431)
0619 18:41:13.654863 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.654918 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.656541 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.657340 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.657350 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.657454 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.658121 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.658124 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.658164 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.658257 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.658717 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.658729 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.658738 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.659058 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.659353 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":439,"threads_started":1}
I20260619 18:41:13.661826 8537 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46483
I20260619 18:41:13.662173 8537 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:13.662873 8537 heartbeater.cc:511] Master 127.3.85.190:46483 requested a full tablet report, sending...
I20260619 18:41:13.664371 8090 ts_manager.cc:194] Registered new tserver with Master: 5417f4cdc9c24e4e84f8b546efef9cfa (127.3.85.131:44635)
I20260619 18:41:13.669745 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:13.677227 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.674279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.3.85.190:46483)
0619 18:41:13.674529 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.674534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.675048 (+ 514us) server_negotiation.cc:407] Connection header received
0619 18:41:13.675262 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.675267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.675336 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.675445 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.676541 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.676547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.676551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.676580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.676614 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.676627 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.676773 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.676885 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.676953 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:41:13.681835 8090 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:13.689070 8090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:13.709622 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.703045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:46253 (local address 127.0.0.1:50580)
0619 18:41:13.704074 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.704137 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.704436 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.706178 (+ 1742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.706186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.706297 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.707795 (+ 1498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.707798 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.707841 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.707944 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.708685 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.708698 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.708707 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.708901 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.709025 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":438,"threads_started":1}
I20260619 18:41:13.710815 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.703135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.3.85.130:46253)
0619 18:41:13.705033 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.705050 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.705081 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:41:13.705153 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.705162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.705418 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.705586 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.708217 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.708231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.708253 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.708290 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.708366 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.708398 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.708555 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.710221 (+ 1666us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.710354 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1649,"thread_start_us":2752,"threads_started":1}
I20260619 18:41:13.710901 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.704502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.3.85.131:44635)
0619 18:41:13.707311 (+ 2809us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.707320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.707604 (+ 284us) server_negotiation.cc:407] Connection header received
0619 18:41:13.707746 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.707759 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.708002 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.708672 (+ 670us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.710083 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.710090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.710101 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.710113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.710156 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.710170 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.710359 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.710430 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.710546 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2615,"thread_start_us":2414,"threads_started":1}
I20260619 18:41:13.714527 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.704238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:44635 (local address 127.0.0.1:53566)
0619 18:41:13.707455 (+ 3217us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.707477 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.707600 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.708218 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.708226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.708255 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.709345 (+ 1090us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.709347 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.709361 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.709401 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.710281 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.710285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.710291 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.710418 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.710514 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3068,"thread_start_us":2939,"threads_started":1}
I20260619 18:41:13.715214 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.707517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54770)
0619 18:41:13.709911 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.709933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.710023 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.713090 (+ 3067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.713100 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.713140 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.713795 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.713800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.713818 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.713979 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.714709 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.714715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.714721 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.714839 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.714904 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2251,"thread_start_us":3663,"threads_started":1}
I20260619 18:41:13.716041 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.707795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.3.85.129:35657)
0619 18:41:13.712160 (+ 4365us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.712169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.712205 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:41:13.712274 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.712282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.712535 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.712702 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.714220 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.714231 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.714247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.714269 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.714335 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.714384 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.714658 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.715533 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.715646 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3887,"thread_start_us":3764,"threads_started":1}
I20260619 18:41:13.716292 8342 tablet_service.cc:1467] Processing CreateTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 (DEFAULT_TABLE table=test-table [id=2e6c6d04b25f4364ac0352ece4d44b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:13.718050 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc3cd7dfaa84121becd86ac65c12916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:13.723044 8211 tablet_service.cc:1467] Processing CreateTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 (DEFAULT_TABLE table=test-table [id=2e6c6d04b25f4364ac0352ece4d44b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:13.723044 8472 tablet_service.cc:1467] Processing CreateTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 (DEFAULT_TABLE table=test-table [id=2e6c6d04b25f4364ac0352ece4d44b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:13.724742 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc3cd7dfaa84121becd86ac65c12916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:13.724766 8472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccc3cd7dfaa84121becd86ac65c12916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:13.735267 8276 heartbeater.cc:503] Master 127.3.85.190:46483 was elected leader, sending a full tablet report...
I20260619 18:41:13.736054 8555 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Bootstrap starting.
I20260619 18:41:13.739477 8556 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: Bootstrap starting.
I20260619 18:41:13.740983 8555 tablet_bootstrap.cc:654] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:13.742086 8555 log.cc:824] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:13.744128 8556 tablet_bootstrap.cc:654] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:13.744266 8558 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Bootstrap starting.
I20260619 18:41:13.745311 8555 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: No bootstrap required, opened a new log
I20260619 18:41:13.745620 8555 ts_tablet_manager.cc:1397] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 18:41:13.745736 8556 log.cc:824] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:13.749648 8558 tablet_bootstrap.cc:654] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:13.751542 8558 log.cc:824] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:13.753069 8556 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: No bootstrap required, opened a new log
I20260619 18:41:13.753407 8556 ts_tablet_manager.cc:1397] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260619 18:41:13.755450 8558 tablet_bootstrap.cc:492] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: No bootstrap required, opened a new log
I20260619 18:41:13.755316 8555 raft_consensus.cc:348] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.755775 8558 ts_tablet_manager.cc:1397] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260619 18:41:13.755838 8555 raft_consensus.cc:374] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:13.755962 8555 raft_consensus.cc:729] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2330627af04059b9744e2e1acd755b, State: Initialized, Role: FOLLOWER
I20260619 18:41:13.756740 8555 consensus_queue.cc:260] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.761513 8556 raft_consensus.cc:348] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.761963 8556 raft_consensus.cc:374] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:13.762081 8556 raft_consensus.cc:729] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ad276c0f484996aac2c61189d91747, State: Initialized, Role: FOLLOWER
I20260619 18:41:13.762895 8407 heartbeater.cc:503] Master 127.3.85.190:46483 was elected leader, sending a full tablet report...
I20260619 18:41:13.762882 8556 consensus_queue.cc:260] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.764478 8555 ts_tablet_manager.cc:1428] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260619 18:41:13.767400 8556 ts_tablet_manager.cc:1428] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 18:41:13.769223 8558 raft_consensus.cc:348] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.769899 8558 raft_consensus.cc:374] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:13.770064 8558 raft_consensus.cc:729] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5417f4cdc9c24e4e84f8b546efef9cfa, State: Initialized, Role: FOLLOWER
I20260619 18:41:13.770987 8558 consensus_queue.cc:260] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.772778 8537 heartbeater.cc:503] Master 127.3.85.190:46483 was elected leader, sending a full tablet report...
I20260619 18:41:13.773515 8558 ts_tablet_manager.cc:1428] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260619 18:41:13.825443 8563 raft_consensus.cc:482] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:13.825840 8563 raft_consensus.cc:504] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.827517 8563 leader_election.cc:288] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d2330627af04059b9744e2e1acd755b (127.3.85.130:46253), d7ad276c0f484996aac2c61189d91747 (127.3.85.129:35657)
I20260619 18:41:13.830621 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.827977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:46253 (local address 127.3.85.131:49991)
0619 18:41:13.828311 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.828336 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.828526 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.828974 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.828983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.829017 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.829738 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.829741 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.829758 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.829800 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.830169 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.830178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.830184 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.830302 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.830367 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260619 18:41:13.830763 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.827977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49991 (local address 127.3.85.130:46253)
0619 18:41:13.828154 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.828159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.828454 (+ 295us) server_negotiation.cc:407] Connection header received
0619 18:41:13.828687 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.828694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.828766 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.828876 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.829969 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.829977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.829985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.830007 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.830045 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.830061 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.830187 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.830297 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.830358 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:41:13.831840 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc3cd7dfaa84121becd86ac65c12916" candidate_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d2330627af04059b9744e2e1acd755b" is_pre_election: true
I20260619 18:41:13.832150 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.828652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:38823 (local address 127.3.85.129:35657)
0619 18:41:13.828859 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.828865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.829201 (+ 336us) server_negotiation.cc:407] Connection header received
0619 18:41:13.829395 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.829401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.829485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.829604 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.831487 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.831493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.831497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.831520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.831564 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.831584 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.831724 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.831805 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.831874 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:41:13.832480 8362 raft_consensus.cc:2454] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5417f4cdc9c24e4e84f8b546efef9cfa in term 0.
I20260619 18:41:13.832564 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.828388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.3.85.131:38823)
0619 18:41:13.829093 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:13.829110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:13.829252 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:13.829627 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:13.829633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:13.829655 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:13.831232 (+ 1577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:13.831235 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:13.831254 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:13.831309 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:13.831683 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:13.831688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:13.831694 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:13.831813 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:41:13.831875 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":421,"threads_started":1}
I20260619 18:41:13.833197 8425 leader_election.cc:302] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5417f4cdc9c24e4e84f8b546efef9cfa, 6d2330627af04059b9744e2e1acd755b; no voters:
I20260619 18:41:13.833662 8563 raft_consensus.cc:2781] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:13.833834 8563 raft_consensus.cc:482] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:13.833949 8563 raft_consensus.cc:3037] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:13.833757 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc3cd7dfaa84121becd86ac65c12916" candidate_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ad276c0f484996aac2c61189d91747" is_pre_election: true
I20260619 18:41:13.834383 8231 raft_consensus.cc:2454] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5417f4cdc9c24e4e84f8b546efef9cfa in term 0.
I20260619 18:41:13.836505 8563 raft_consensus.cc:504] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.837105 8563 leader_election.cc:288] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [CANDIDATE]: Term 1 election: Requested vote from peers 6d2330627af04059b9744e2e1acd755b (127.3.85.130:46253), d7ad276c0f484996aac2c61189d91747 (127.3.85.129:35657)
I20260619 18:41:13.837771 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc3cd7dfaa84121becd86ac65c12916" candidate_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d2330627af04059b9744e2e1acd755b"
I20260619 18:41:13.837954 8231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccc3cd7dfaa84121becd86ac65c12916" candidate_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ad276c0f484996aac2c61189d91747"
I20260619 18:41:13.838018 8362 raft_consensus.cc:3037] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:13.838178 8231 raft_consensus.cc:3037] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:13.841523 8362 raft_consensus.cc:2454] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5417f4cdc9c24e4e84f8b546efef9cfa in term 1.
I20260619 18:41:13.841523 8231 raft_consensus.cc:2454] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5417f4cdc9c24e4e84f8b546efef9cfa in term 1.
I20260619 18:41:13.842077 8425 leader_election.cc:302] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5417f4cdc9c24e4e84f8b546efef9cfa, 6d2330627af04059b9744e2e1acd755b; no voters:
I20260619 18:41:13.842581 8563 raft_consensus.cc:2781] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:13.843034 8563 raft_consensus.cc:686] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 1 LEADER]: Becoming Leader. State: Replica: 5417f4cdc9c24e4e84f8b546efef9cfa, State: Running, Role: LEADER
I20260619 18:41:13.843540 8563 consensus_queue.cc:237] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } }
I20260619 18:41:13.848475 8089 catalog_manager.cc:5495] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5417f4cdc9c24e4e84f8b546efef9cfa (127.3.85.131). New cstate: current_term: 1 leader_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5417f4cdc9c24e4e84f8b546efef9cfa" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 44635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:13.871693 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.869556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.3.85.131:44635)
0619 18:41:13.869714 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.869719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.869736 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:41:13.869855 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.869858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.869937 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.870031 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.871134 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.871138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.871142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.871158 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.871189 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.871202 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.871328 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.871452 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.871502 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:41:13.880446 8362 raft_consensus.cc:1261] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 1 FOLLOWER]: Refusing update from remote peer 5417f4cdc9c24e4e84f8b546efef9cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:13.880456 8231 raft_consensus.cc:1261] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 1 FOLLOWER]: Refusing update from remote peer 5417f4cdc9c24e4e84f8b546efef9cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:13.881645 8565 consensus_queue.cc:1035] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d2330627af04059b9744e2e1acd755b" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:13.882141 8563 consensus_queue.cc:1035] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ad276c0f484996aac2c61189d91747" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:13.896060 8570 mvcc.cc:205] Tried to move back new op lower bound from 7298639765003837440 to 7298639764867788800. Current Snapshot: MvccSnapshot[applied={T|T < 7298639765003837440}]
I20260619 18:41:13.920513 8090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:13.920826 8090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:13.924893 8090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fc5e026039645a78f21289a2793ce2a: Sending DeleteTablet for 3 replicas of tablet ccc3cd7dfaa84121becd86ac65c12916
I20260619 18:41:13.926275 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:13 UTC) from {username='slave'} at 127.0.0.1:50580
I20260619 18:41:13.926312 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:13 UTC) from {username='slave'} at 127.0.0.1:54770
I20260619 18:41:13.926614 8472 tablet_service.cc:1514] Processing DeleteTablet for tablet ccc3cd7dfaa84121becd86ac65c12916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:13 UTC) from {username='slave'} at 127.0.0.1:53566
I20260619 18:41:13.930231 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8149
I20260619 18:41:13.938831 8581 tablet_replica.cc:330] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: stopping tablet replica
I20260619 18:41:13.939677 8581 raft_consensus.cc:2229] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:13.939957 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:13.932186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.3.85.129:35657)
0619 18:41:13.932426 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:13.932436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:13.935397 (+ 2961us) server_negotiation.cc:407] Connection header received
0619 18:41:13.935650 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:13.935658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:13.935827 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:13.935981 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:13.938589 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:13.938598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:13.938604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:13.938635 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:13.938686 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:13.938710 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:13.938864 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:13.939447 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 18:41:13.939530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:41:13.941044 8581 raft_consensus.cc:2258] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:13.941322 8271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:13.942051 8580 tablet_replica.cc:330] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: stopping tablet replica
I20260619 18:41:13.944631 8581 ts_tablet_manager.cc:1905] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:13.945392 8580 raft_consensus.cc:2229] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:13.947829 8580 raft_consensus.cc:2258] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:13.956861 8580 ts_tablet_manager.cc:1905] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:13.956935 8581 ts_tablet_manager.cc:1918] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:13.957294 8581 log.cc:1196] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/ccc3cd7dfaa84121becd86ac65c12916
I20260619 18:41:13.957976 8581 ts_tablet_manager.cc:1939] T ccc3cd7dfaa84121becd86ac65c12916 P 5417f4cdc9c24e4e84f8b546efef9cfa: Deleting consensus metadata
I20260619 18:41:13.960052 8074 catalog_manager.cc:4841] TS 5417f4cdc9c24e4e84f8b546efef9cfa (127.3.85.131:44635): tablet ccc3cd7dfaa84121becd86ac65c12916 (table test-table [id=2e6c6d04b25f4364ac0352ece4d44b28]) successfully deleted
I20260619 18:41:13.963400 8580 ts_tablet_manager.cc:1918] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:13.963610 8580 log.cc:1196] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/ccc3cd7dfaa84121becd86ac65c12916
I20260619 18:41:13.964061 8580 ts_tablet_manager.cc:1939] T ccc3cd7dfaa84121becd86ac65c12916 P 6d2330627af04059b9744e2e1acd755b: Deleting consensus metadata
I20260619 18:41:13.965763 8074 catalog_manager.cc:4841] TS 6d2330627af04059b9744e2e1acd755b (127.3.85.130:46253): tablet ccc3cd7dfaa84121becd86ac65c12916 (table test-table [id=2e6c6d04b25f4364ac0352ece4d44b28]) successfully deleted
I20260619 18:41:14.053610 8579 tablet_replica.cc:330] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747: stopping tablet replica
I20260619 18:41:14.054436 8579 raft_consensus.cc:2229] T ccc3cd7dfaa84121becd86ac65c12916 P d7ad276c0f484996aac2c61189d91747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:14.054581 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8149
W20260619 18:41:14.071967 8074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:35657 (error 108)
I20260619 18:41:14.072196 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8280
I20260619 18:41:14.074441 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.073875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54798)
0619 18:41:14.074072 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.074223 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:41:14.074781 8074 catalog_manager.cc:4584] TS d7ad276c0f484996aac2c61189d91747 (127.3.85.129:35657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccc3cd7dfaa84121becd86ac65c12916: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
I20260619 18:41:14.075569 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.073135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.3.85.130:46253)
0619 18:41:14.073346 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:14.073353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:14.073380 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:41:14.073512 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:14.073521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:14.073639 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:14.073784 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:14.074813 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:14.074820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:14.074824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:14.074850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:14.074881 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:14.074893 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:14.075027 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:14.075161 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 18:41:14.075236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:41:14.076272 8402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:14.128788 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.128202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54814)
0619 18:41:14.128486 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.128606 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 18:41:14.130111 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.129634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54818)
0619 18:41:14.129825 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.129931 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:41:14.176738 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8280
I20260619 18:41:14.190773 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.190274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54822)
0619 18:41:14.190521 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.190629 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:41:14.192035 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.191608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54824)
0619 18:41:14.191784 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.191883 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:41:14.199360 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8410
I20260619 18:41:14.202598 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.200212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.3.85.131:44635)
0619 18:41:14.200395 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:14.200399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:14.200471 (+ 72us) server_negotiation.cc:407] Connection header received
0619 18:41:14.200685 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:14.200690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:14.200756 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:14.200846 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:14.201823 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:14.201828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:14.201831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:14.201847 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:14.201878 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:14.201890 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:14.202019 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:14.202136 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:41:14.202185 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:41:14.203442 8532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:14.273101 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.272202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54832)
0619 18:41:14.272487 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.272844 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 18:41:14.274672 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.274091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:35657 (local address 127.0.0.1:54848)
0619 18:41:14.274312 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.274408 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 18:41:14.319681 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8410
I20260619 18:41:14.338027 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8059
I20260619 18:41:14.339324 8120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:14.444538 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8059
2026-06-19T18:41:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2793 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 18:41:14.486773 3414 mini_oidc.cc:152] Starting JWKS server
I20260619 18:41:14.489444 3414 webserver.cc:460] Webserver started at https://127.0.0.1:36657/ using document root <none> and password file <none>
I20260619 18:41:14.489825 3414 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:41:14.490598 3414 webserver.cc:460] Webserver started at http://127.0.0.1:43047/ using document root <none> and password file <none>
2026-06-19T18:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:14Z Disabled control of system clock
I20260619 18:41:14.519980 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39803
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39803
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36657/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/testchainca.cert with env {}
W20260619 18:41:14.863608 8593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:14.863953 8593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:14.864044 8593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:14.871587 8593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:14.871713 8593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:14.871752 8593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:14.871802 8593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:14.871847 8593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:41:14.871917 8593 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:41:14.871958 8593 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36657/jwks.json
I20260619 18:41:14.882161 8593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36657/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:14.883577 8593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:14.884861 8593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:14.897207 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.897249 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.897964 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.898801 8593 server_base.cc:1034] running on GCE node
I20260619 18:41:14.899710 8593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:14.901419 8593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:14.902680 8593 hybrid_clock.cc:648] HybridClock initialized: now 1781894474902613 us; error 77 us; skew 500 ppm
I20260619 18:41:14.903079 8593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:14.905840 8593 webserver.cc:460] Webserver started at http://127.3.85.190:34333/ using document root <none> and password file <none>
I20260619 18:41:14.906819 8593 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:14.906973 8593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:14.907310 8593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:14.910501 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "57b78ea61f594abfa9bf5a1d11c4f660"
format_stamp: "Formatted at 2026-06-19 18:41:14 on dist-test-slave-09fp"
server_key: "996b732b83d96d5b83c771ebc334830f"
server_key_iv: "aee2ec55161a520b7ea181d55c0e5d13"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.911417 8593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "57b78ea61f594abfa9bf5a1d11c4f660"
format_stamp: "Formatted at 2026-06-19 18:41:14 on dist-test-slave-09fp"
server_key: "996b732b83d96d5b83c771ebc334830f"
server_key_iv: "aee2ec55161a520b7ea181d55c0e5d13"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.917104 8593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:41:14.920972 8607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.922653 8593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:41:14.922884 8593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "57b78ea61f594abfa9bf5a1d11c4f660"
format_stamp: "Formatted at 2026-06-19 18:41:14 on dist-test-slave-09fp"
server_key: "996b732b83d96d5b83c771ebc334830f"
server_key_iv: "aee2ec55161a520b7ea181d55c0e5d13"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.923138 8593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:14.941942 8593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:14.943158 8593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:14.943514 8593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:14.963433 8593 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39803
I20260619 18:41:14.963450 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39803 every 8 connection(s)
I20260619 18:41:14.964941 8593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:14.969036 8659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:14.971026 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8593
I20260619 18:41:14.971252 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:14.971884 3414 external_mini_cluster.cc:1428] Setting key b3415901a9f34771a9ed5bc1e91ea925
I20260619 18:41:14.981509 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660: Bootstrap starting.
I20260619 18:41:14.985201 8659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:14.986382 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.973987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.3.85.190:39803)
0619 18:41:14.975157 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:14.975173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:14.975223 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:41:14.977009 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:14.977035 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:14.977562 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:14.978009 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:14.978689 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:14.980036 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:14.981472 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:14.982255 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:14.982512 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:14.983885 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:14.983893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:14.983925 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:14.984490 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:14.984584 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:14.984605 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:14.984746 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:14.984989 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 18:41:14.985357 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":430,"threads_started":1}
I20260619 18:41:14.986389 8659 log.cc:824] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:14.989670 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660: No bootstrap required, opened a new log
I20260619 18:41:14.990690 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:14.998669 8659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } }
I20260619 18:41:14.999154 8659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:14.999279 8659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b78ea61f594abfa9bf5a1d11c4f660, State: Initialized, Role: FOLLOWER
I20260619 18:41:14.999915 8659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } }
I20260619 18:41:15.000196 8659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:15.000367 8659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:15.000567 8659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:15.002988 8659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } }
I20260619 18:41:15.003530 8659 leader_election.cc:302] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b78ea61f594abfa9bf5a1d11c4f660; no voters:
I20260619 18:41:15.004108 8659 leader_election.cc:288] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:15.004312 8664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:15.005090 8664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [term 1 LEADER]: Becoming Leader. State: Replica: 57b78ea61f594abfa9bf5a1d11c4f660, State: Running, Role: LEADER
I20260619 18:41:15.005630 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } }
I20260619 18:41:15.006510 8659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:15.008566 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } } }
I20260619 18:41:15.009053 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:15.009262 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b78ea61f594abfa9bf5a1d11c4f660. Latest consensus state: current_term: 1 leader_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b78ea61f594abfa9bf5a1d11c4f660" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39803 } } }
I20260619 18:41:15.009735 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:15.013790 8669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:15.020076 8669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:15.029552 8669 catalog_manager.cc:1347] Generated new cluster ID: e3be8cf1759b4dcc9677fa645d085204
I20260619 18:41:15.029693 8669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:15.042313 8669 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:15.043743 8669 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:15.050830 8669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57b78ea61f594abfa9bf5a1d11c4f660: Generated new TSK 0
I20260619 18:41:15.051800 8669 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:41:15.092743 8689 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 18:41:15.092929 8689 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:41:15.094162 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.072151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.3.85.190:39803)
0619 18:41:15.072374 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:15.072383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:15.072887 (+ 504us) server_negotiation.cc:407] Connection header received
0619 18:41:15.073186 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:15.073192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:15.073288 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:15.073531 (+ 243us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:41:15.074224 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.075437 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.076956 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.077684 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.077828 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.078407 (+ 579us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:41:15.093377 (+ 14970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:41:15.093841 (+ 464us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:41:15.094293 8662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 18:41:15.095711 8688 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:15.071968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39803 (local address 127.0.0.1:58922)
0619 18:41:15.072770 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.072787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:15.072946 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:15.073670 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:15.073674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:15.073685 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:41:15.074015 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.074024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.075621 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.075626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.076790 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.076803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.077891 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.077895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.078032 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.078103 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:41:15.093740 (+ 15637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:41:15.094087 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.85.190:39803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":445,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.85.190:39803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 18:41:15.097731 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8593
I20260619 18:41:15.099474 8654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:15.193182 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8593
2026-06-19T18:41:15Z chronyd exiting
I20260619 18:41:15.241735 3414 test_util.cc:175] -----------------------------------------------
I20260619 18:41:15.241931 3414 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894429649633-3414-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 18:41:15.245829 3414 mini_oidc.cc:152] Starting JWKS server
I20260619 18:41:15.247844 3414 webserver.cc:460] Webserver started at https://127.0.0.1:40995/ using document root <none> and password file <none>
I20260619 18:41:15.248114 3414 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:41:15.248816 3414 webserver.cc:460] Webserver started at http://127.0.0.1:43477/ using document root <none> and password file <none>
2026-06-19T18:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:15Z Disabled control of system clock
I20260619 18:41:15.277997 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:33237
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:33237
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40995/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/testchainca.cert with env {}
W20260619 18:41:15.625967 8696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:15.626319 8696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:15.626523 8696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:15.634294 8696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:15.634507 8696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:15.634591 8696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:15.634694 8696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:15.634784 8696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:41:15.634891 8696 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:41:15.634979 8696 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40995/jwks.json
I20260619 18:41:15.645852 8696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:33237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:33237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40995/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:15.647359 8696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:15.648824 8696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:15.662490 8702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:15.662643 8704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:15.662608 8701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:15.663455 8696 server_base.cc:1034] running on GCE node
I20260619 18:41:15.664407 8696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:15.666169 8696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:15.667472 8696 hybrid_clock.cc:648] HybridClock initialized: now 1781894475667430 us; error 67 us; skew 500 ppm
I20260619 18:41:15.667934 8696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:15.675544 8696 webserver.cc:460] Webserver started at http://127.3.85.190:44743/ using document root <none> and password file <none>
I20260619 18:41:15.676425 8696 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:15.676586 8696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:15.676975 8696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:15.680213 8696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "35b3200a1dd24470ad6a1322e6f3770f"
format_stamp: "Formatted at 2026-06-19 18:41:15 on dist-test-slave-09fp"
server_key: "e4539f319b3e2e30683c348fed5e2585"
server_key_iv: "1f86a29e587e62393cd1c0f0a84675bf"
server_key_version: "encryptionkey@0"
I20260619 18:41:15.681118 8696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "35b3200a1dd24470ad6a1322e6f3770f"
format_stamp: "Formatted at 2026-06-19 18:41:15 on dist-test-slave-09fp"
server_key: "e4539f319b3e2e30683c348fed5e2585"
server_key_iv: "1f86a29e587e62393cd1c0f0a84675bf"
server_key_version: "encryptionkey@0"
I20260619 18:41:15.686952 8696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:41:15.690896 8710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:15.692611 8696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:15.692866 8696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "35b3200a1dd24470ad6a1322e6f3770f"
format_stamp: "Formatted at 2026-06-19 18:41:15 on dist-test-slave-09fp"
server_key: "e4539f319b3e2e30683c348fed5e2585"
server_key_iv: "1f86a29e587e62393cd1c0f0a84675bf"
server_key_version: "encryptionkey@0"
I20260619 18:41:15.693145 8696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:15.708340 8696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:15.709263 8696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:15.709601 8696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:15.729979 8696 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:33237
I20260619 18:41:15.729998 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:33237 every 8 connection(s)
I20260619 18:41:15.731527 8696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:15.735503 8762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:15.739841 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8696
I20260619 18:41:15.740047 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:15.740680 3414 external_mini_cluster.cc:1428] Setting key ce79b51bb114041a42161ea5c7740faf
I20260619 18:41:15.748900 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f: Bootstrap starting.
I20260619 18:41:15.752910 8762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:15.754390 8762 log.cc:824] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:15.757889 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.742844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.3.85.190:33237)
0619 18:41:15.743999 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:15.744015 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:15.744071 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:41:15.746761 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:15.746796 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:15.747482 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:15.747953 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:15.748573 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.750615 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.752177 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.753405 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.753631 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.754948 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:15.754961 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:15.754986 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:15.755712 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:15.755802 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:15.755827 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:15.755995 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:15.756223 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 18:41:15.756587 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":356,"threads_started":1}
I20260619 18:41:15.758186 8762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f: No bootstrap required, opened a new log
I20260619 18:41:15.762377 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:15.767046 8762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } }
I20260619 18:41:15.767491 8762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:15.767623 8762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b3200a1dd24470ad6a1322e6f3770f, State: Initialized, Role: FOLLOWER
I20260619 18:41:15.768262 8762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } }
I20260619 18:41:15.768517 8762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:15.768649 8762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:15.768826 8762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:15.771354 8762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } }
I20260619 18:41:15.772118 8762 leader_election.cc:302] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35b3200a1dd24470ad6a1322e6f3770f; no voters:
I20260619 18:41:15.772786 8762 leader_election.cc:288] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:15.772904 8767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:15.773648 8767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [term 1 LEADER]: Becoming Leader. State: Replica: 35b3200a1dd24470ad6a1322e6f3770f, State: Running, Role: LEADER
I20260619 18:41:15.774282 8767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } }
I20260619 18:41:15.775199 8762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:15.777168 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35b3200a1dd24470ad6a1322e6f3770f. Latest consensus state: current_term: 1 leader_uuid: "35b3200a1dd24470ad6a1322e6f3770f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } } }
I20260619 18:41:15.777613 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:15.778533 8768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35b3200a1dd24470ad6a1322e6f3770f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35b3200a1dd24470ad6a1322e6f3770f" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 33237 } } }
I20260619 18:41:15.778733 8774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:15.778882 8768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:15.784897 8774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:15.795984 8774 catalog_manager.cc:1347] Generated new cluster ID: 30101b9bb2534bbcb878b0de7a012699
I20260619 18:41:15.796139 8774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:15.813510 8774 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:15.814980 8774 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:15.829345 8774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35b3200a1dd24470ad6a1322e6f3770f: Generated new TSK 0
I20260619 18:41:15.830267 8774 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:41:15.876264 8792 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 18:41:15.876400 8792 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 18:41:15.877930 8791 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:15.856180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:33237 (local address 127.0.0.1:59052)
0619 18:41:15.856804 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.856821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:15.856965 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:15.857534 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:15.857537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:15.857548 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:41:15.857792 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.857801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.859398 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.859403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.860511 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.860524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.861622 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.861627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.861780 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.861818 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:41:15.877399 (+ 15581us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:41:15.877625 (+ 226us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.85.190:33237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
I20260619 18:41:15.877928 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.855953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.3.85.190:33237)
0619 18:41:15.856163 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:15.856168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:15.856939 (+ 771us) server_negotiation.cc:407] Connection header received
0619 18:41:15.857197 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:15.857204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:15.857313 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:15.857507 (+ 194us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:41:15.857953 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.859187 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.860681 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.861417 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.861572 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.862237 (+ 665us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:41:15.877080 (+ 14843us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:41:15.877563 (+ 483us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 18:41:15.878067 8765 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 18:41:15.878759 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8696
I20260619 18:41:15.880503 8757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:15.976595 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8696
2026-06-19T18:41:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (787 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 18:41:16.033914 3414 mini_oidc.cc:152] Starting JWKS server
I20260619 18:41:16.035948 3414 webserver.cc:460] Webserver started at https://127.0.0.1:32907/ using document root <none> and password file <none>
I20260619 18:41:16.036250 3414 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:41:16.036988 3414 webserver.cc:460] Webserver started at http://127.0.0.1:32881/ using document root <none> and password file <none>
2026-06-19T18:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:16Z Disabled control of system clock
I20260619 18:41:16.066017 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43811
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:42097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:43811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:41:16.418970 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:16.419394 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:16.419574 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:16.427474 8799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:16.427680 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:16.427786 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:16.427918 8799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:16.428004 8799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:41:16.428138 8799 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:41:16.428222 8799 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32907/jwks.json
I20260619 18:41:16.438673 8799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:43811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:16.440131 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:16.442140 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:16.455989 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:16.456147 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:16.456020 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:16.457118 8799 server_base.cc:1034] running on GCE node
I20260619 18:41:16.457988 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:16.459700 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:16.460981 8799 hybrid_clock.cc:648] HybridClock initialized: now 1781894476460942 us; error 66 us; skew 500 ppm
I20260619 18:41:16.461413 8799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:16.464219 8799 webserver.cc:460] Webserver started at http://127.3.85.190:36849/ using document root <none> and password file <none>
I20260619 18:41:16.465046 8799 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:16.465202 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:16.465581 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:16.468966 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "a6c2357d9e8e45c687c1f32c30e64e8e"
format_stamp: "Formatted at 2026-06-19 18:41:16 on dist-test-slave-09fp"
server_key: "578e863a6a273c70b43de612f7e13a00"
server_key_iv: "78b6aa375139f116b745926971f914c4"
server_key_version: "encryptionkey@0"
I20260619 18:41:16.469976 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "a6c2357d9e8e45c687c1f32c30e64e8e"
format_stamp: "Formatted at 2026-06-19 18:41:16 on dist-test-slave-09fp"
server_key: "578e863a6a273c70b43de612f7e13a00"
server_key_iv: "78b6aa375139f116b745926971f914c4"
server_key_version: "encryptionkey@0"
I20260619 18:41:16.476076 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:41:16.480247 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:16.482115 8799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:41:16.482374 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "a6c2357d9e8e45c687c1f32c30e64e8e"
format_stamp: "Formatted at 2026-06-19 18:41:16 on dist-test-slave-09fp"
server_key: "578e863a6a273c70b43de612f7e13a00"
server_key_iv: "78b6aa375139f116b745926971f914c4"
server_key_version: "encryptionkey@0"
I20260619 18:41:16.482744 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:16.510618 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:16.511543 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:16.511907 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:16.531970 8799 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:43811
I20260619 18:41:16.531989 8864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:43811 every 8 connection(s)
I20260619 18:41:16.533456 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:16.538110 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8799
I20260619 18:41:16.538301 8865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:16.538352 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:16.539286 3414 external_mini_cluster.cc:1428] Setting key 7da4ac10400d165a9e17cc38ddcb102a
I20260619 18:41:16.552898 8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e: Bootstrap starting.
I20260619 18:41:16.555070 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:16.541440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.3.85.190:43811)
0619 18:41:16.542767 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:16.542779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:16.542833 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:41:16.545297 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:16.545326 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:16.545913 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:16.546376 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:16.547005 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.548292 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.550128 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.551022 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.551251 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:16.552474 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:16.552486 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:16.552502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:16.553124 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:16.553222 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:16.553246 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:16.553380 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:16.553618 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 18:41:16.553972 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":469,"threads_started":1}
I20260619 18:41:16.556774 8865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:16.558025 8865 log.cc:824] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:16.559310 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:16.561654 8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e: No bootstrap required, opened a new log
I20260619 18:41:16.570196 8865 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } }
I20260619 18:41:16.570771 8865 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:16.570897 8865 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c2357d9e8e45c687c1f32c30e64e8e, State: Initialized, Role: FOLLOWER
I20260619 18:41:16.571594 8865 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } }
I20260619 18:41:16.571936 8865 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:16.572124 8865 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:16.572361 8865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:16.575028 8865 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } }
I20260619 18:41:16.575652 8865 leader_election.cc:302] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c2357d9e8e45c687c1f32c30e64e8e; no voters:
I20260619 18:41:16.576264 8865 leader_election.cc:288] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:16.576463 8870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:16.577469 8870 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [term 1 LEADER]: Becoming Leader. State: Replica: a6c2357d9e8e45c687c1f32c30e64e8e, State: Running, Role: LEADER
I20260619 18:41:16.578001 8870 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } }
I20260619 18:41:16.578797 8865 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:16.581357 8871 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } } }
I20260619 18:41:16.581331 8872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c2357d9e8e45c687c1f32c30e64e8e. Latest consensus state: current_term: 1 leader_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c2357d9e8e45c687c1f32c30e64e8e" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43811 } } }
I20260619 18:41:16.581864 8871 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:16.581893 8872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:16.582851 8876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:16.593982 8876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:16.604458 8876 catalog_manager.cc:1347] Generated new cluster ID: 0bbbfe8966bf40a58e3cfff8d98ea351
I20260619 18:41:16.604599 8876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:16.618171 8876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:16.620158 8876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:16.628818 8876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c2357d9e8e45c687c1f32c30e64e8e: Generated new TSK 0
I20260619 18:41:16.630033 8876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:16.655391 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:16.646313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.3.85.190:43811)
0619 18:41:16.646567 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:16.646575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:16.647021 (+ 446us) server_negotiation.cc:407] Connection header received
0619 18:41:16.647235 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:16.647242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:16.647346 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:16.647554 (+ 208us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:41:16.648001 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.649201 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.650710 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.651358 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.651513 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:16.652094 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:41:16.654682 (+ 2588us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:41:16.655060 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 18:41:16.655529 8868 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 18:41:16.657187 8894 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:16.646153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43811 (local address 127.0.0.1:50890)
0619 18:41:16.646900 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:16.646917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:16.647055 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:16.647564 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:16.647567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:16.647577 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:41:16.647844 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:16.647853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.649374 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.649379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:16.650542 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:16.650556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:16.651567 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:16.651572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:16.651731 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:16.651771 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:41:16.654953 (+ 3182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:41:16.656753 (+ 1800us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.85.190:43811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":1434,"thread_start_us":414,"threads_started":1}
I20260619 18:41:16.659307 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8799
I20260619 18:41:16.660657 8860 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 18:41:16.670549 8895 webserver.cc:533] Webserver: sslize failed:
E20260619 18:41:16.670686 8895 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:41:16.758608 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8799
2026-06-19T18:41:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (781 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T18:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:16Z Disabled control of system clock
I20260619 18:41:16.844902 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:45277
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:35051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:45277
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/testchainca.cert with env {}
W20260619 18:41:17.193264 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:17.193604 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:17.193765 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:17.201361 8900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:17.201560 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:17.201640 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:17.201743 8900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:17.201824 8900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:41:17.201921 8900 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:41:17.201967 8900 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 18:41:17.212517 8900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:35051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:45277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:17.213924 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:17.215366 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:17.227975 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:17.228044 8905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:17.227993 8906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:17.228806 8900 server_base.cc:1034] running on GCE node
I20260619 18:41:17.229614 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:17.231370 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:17.232621 8900 hybrid_clock.cc:648] HybridClock initialized: now 1781894477232556 us; error 71 us; skew 500 ppm
I20260619 18:41:17.233013 8900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:17.235579 8900 webserver.cc:460] Webserver started at http://127.3.85.190:33239/ using document root <none> and password file <none>
I20260619 18:41:17.236342 8900 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:17.236486 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:17.236819 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:17.239921 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "21280f57e96d4a978f87c4847c82386c"
format_stamp: "Formatted at 2026-06-19 18:41:17 on dist-test-slave-09fp"
server_key: "ab0c2aa39c251095e17f5ce8ead6f9c1"
server_key_iv: "108ba48036b73ec5c415ab2784987a2e"
server_key_version: "encryptionkey@0"
I20260619 18:41:17.240815 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "21280f57e96d4a978f87c4847c82386c"
format_stamp: "Formatted at 2026-06-19 18:41:17 on dist-test-slave-09fp"
server_key: "ab0c2aa39c251095e17f5ce8ead6f9c1"
server_key_iv: "108ba48036b73ec5c415ab2784987a2e"
server_key_version: "encryptionkey@0"
I20260619 18:41:17.246769 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:41:17.250960 8914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:17.252530 8900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:17.252767 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "21280f57e96d4a978f87c4847c82386c"
format_stamp: "Formatted at 2026-06-19 18:41:17 on dist-test-slave-09fp"
server_key: "ab0c2aa39c251095e17f5ce8ead6f9c1"
server_key_iv: "108ba48036b73ec5c415ab2784987a2e"
server_key_version: "encryptionkey@0"
I20260619 18:41:17.253026 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:17.271795 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:17.272789 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:17.273106 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:17.292538 8900 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:45277
I20260619 18:41:17.292637 8965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:45277 every 8 connection(s)
I20260619 18:41:17.294018 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:17.297950 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 8900
I20260619 18:41:17.298152 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:17.298408 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:17.299034 3414 external_mini_cluster.cc:1428] Setting key 81260089b60f3abfcb5576c2c0fcd3eb
I20260619 18:41:17.312907 8966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c: Bootstrap starting.
I20260619 18:41:17.314903 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:17.301085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.3.85.190:45277)
0619 18:41:17.302170 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:17.302181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:17.302226 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:17.304777 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:17.304804 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:17.305361 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:17.305744 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:17.306374 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.307753 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.309820 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.310709 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.310865 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:17.312186 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:17.312194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:17.312211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:17.312897 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:17.312983 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:17.313007 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:17.313145 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:17.313449 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 18:41:17.313806 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":388,"threads_started":1}
I20260619 18:41:17.316855 8966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:17.318114 8966 log.cc:824] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:17.319177 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:17.321249 8966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c: No bootstrap required, opened a new log
I20260619 18:41:17.328859 8966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } }
I20260619 18:41:17.329355 8966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:17.329476 8966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21280f57e96d4a978f87c4847c82386c, State: Initialized, Role: FOLLOWER
I20260619 18:41:17.330171 8966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } }
I20260619 18:41:17.330508 8966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:17.330668 8966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:17.330855 8966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:17.333099 8966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } }
I20260619 18:41:17.333738 8966 leader_election.cc:302] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21280f57e96d4a978f87c4847c82386c; no voters:
I20260619 18:41:17.334476 8966 leader_election.cc:288] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:17.334754 8971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:17.335505 8971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [term 1 LEADER]: Becoming Leader. State: Replica: 21280f57e96d4a978f87c4847c82386c, State: Running, Role: LEADER
I20260619 18:41:17.336264 8971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } }
I20260619 18:41:17.337039 8966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:17.339442 8972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21280f57e96d4a978f87c4847c82386c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } } }
I20260619 18:41:17.339385 8973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21280f57e96d4a978f87c4847c82386c. Latest consensus state: current_term: 1 leader_uuid: "21280f57e96d4a978f87c4847c82386c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21280f57e96d4a978f87c4847c82386c" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45277 } } }
I20260619 18:41:17.339864 8972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:17.339916 8973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:17.342859 8978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:17.349306 8978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:17.359633 8978 catalog_manager.cc:1347] Generated new cluster ID: d065cb18b4d64498b6970dafaf2601be
I20260619 18:41:17.359799 8978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:17.369750 8978 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:17.371863 8978 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:17.381121 8978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21280f57e96d4a978f87c4847c82386c: Generated new TSK 0
I20260619 18:41:17.382143 8978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:17.486680 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:17.404574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.3.85.190:45277)
0619 18:41:17.404753 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:17.404758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:17.405136 (+ 378us) server_negotiation.cc:407] Connection header received
0619 18:41:17.405363 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:17.405369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:17.405486 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:17.405682 (+ 196us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:41:17.406187 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.407406 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.408940 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.409628 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.409766 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:17.410341 (+ 575us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:41:17.485885 (+ 75544us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:41:17.486310 (+ 425us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51294: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 18:41:17.487030 8995 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:17.404399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45277 (local address 127.0.0.1:51294)
0619 18:41:17.405022 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:17.405039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:17.405179 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:17.405690 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:17.405694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:17.405704 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:41:17.406024 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:17.406033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.407602 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.407606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:17.408782 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:17.408794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:17.409806 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:17.409810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:17.409932 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:17.409970 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:41:17.486217 (+ 76247us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:41:17.486559 (+ 342us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.85.190:45277: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":335,"threads_started":1}
W20260619 18:41:17.487049 8969 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51294: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 18:41:17.488432 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 8900
I20260619 18:41:17.490315 8961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:17.592401 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 8900
2026-06-19T18:41:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (813 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (442 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (424 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:18 dist-test-slave-09fp krb5kdc[9010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:18 dist-test-slave-09fp krb5kdc[9010](info): set up 2 sockets
Jun 19 18:41:18 dist-test-slave-09fp krb5kdc[9010](info): commencing operation
krb5kdc: starting...
W20260619 18:41:19.575491 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:19 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:19.806351 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35527
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:20.145752 9023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:20.146064 9023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:20.146243 9023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:20.153725 9023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:20.153856 9023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:20.153949 9023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:20.154038 9023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:20.154143 9023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:20.164458 9023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:35527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:20.165966 9023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:20.168053 9023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:20.181047 9028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:20.180990 9031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:20.180902 9029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:20.183784 9023 server_base.cc:1034] running on GCE node
I20260619 18:41:20.184696 9023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:20.186272 9023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:20.187608 9023 hybrid_clock.cc:648] HybridClock initialized: now 1781894480187576 us; error 103 us; skew 500 ppm
I20260619 18:41:20.187965 9023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:20 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:20.191673 9023 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:20.193516 9023 webserver.cc:460] Webserver started at http://127.3.85.190:37035/ using document root <none> and password file <none>
I20260619 18:41:20.194303 9023 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:20.194453 9023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:20.194839 9023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:20.198053 9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "681ac362c03945bb925e1648a35609c8"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "508e1cdc31b153632de4054b15d47a60"
server_key_iv: "492ca840ce22016b651ead4557f9a1d9"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.198940 9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "681ac362c03945bb925e1648a35609c8"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "508e1cdc31b153632de4054b15d47a60"
server_key_iv: "492ca840ce22016b651ead4557f9a1d9"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.205159 9023 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260619 18:41:20.209651 9038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:20.211736 9023 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:41:20.212019 9023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "681ac362c03945bb925e1648a35609c8"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "508e1cdc31b153632de4054b15d47a60"
server_key_iv: "492ca840ce22016b651ead4557f9a1d9"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.212275 9023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:20.231084 9023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:20.236310 9023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:20.236781 9023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:20.260609 9023 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:35527
I20260619 18:41:20.260629 9089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:35527 every 8 connection(s)
I20260619 18:41:20.262257 9023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:20.266642 9090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:20.268497 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9023
I20260619 18:41:20.268716 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:20.269331 3414 external_mini_cluster.cc:1428] Setting key 7aa436f61b9b794907ce2f613ffe504a
I20260619 18:41:20.280535 9090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8: Bootstrap starting.
I20260619 18:41:20.284392 9090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:20.285563 9090 log.cc:824] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:41:20 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:20.288964 9090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8: No bootstrap required, opened a new log
I20260619 18:41:20.296043 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.271541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.3.85.190:35527)
0619 18:41:20.272455 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:20.272479 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:20.272541 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:41:20.275052 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:20.275085 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:20.275641 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:20.276121 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:20.277505 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.279607 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.280959 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.281995 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.282187 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:20.287548 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:20.287558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:20.287575 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:20.287615 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:20.291090 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:20.292132 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:20.292142 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:20.292149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:20.292362 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:20.292810 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:20.292814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:20.292816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:20.293458 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:20.293694 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:20.294220 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 18:41:20.294779 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":380,"threads_started":1}
I20260619 18:41:20.298179 9090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } }
I20260619 18:41:20.298652 9090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:20.298781 9090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681ac362c03945bb925e1648a35609c8, State: Initialized, Role: FOLLOWER
I20260619 18:41:20.299474 9090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } }
I20260619 18:41:20.299751 9090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:20.299923 9090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:20.300097 9090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:20.302758 9090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } }
I20260619 18:41:20.303395 9090 leader_election.cc:302] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 681ac362c03945bb925e1648a35609c8; no voters:
I20260619 18:41:20.304018 9090 leader_election.cc:288] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:20.304165 9095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:20.304896 9095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [term 1 LEADER]: Becoming Leader. State: Replica: 681ac362c03945bb925e1648a35609c8, State: Running, Role: LEADER
I20260619 18:41:20.305568 9095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } }
I20260619 18:41:20.306306 9090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:20.308561 9096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "681ac362c03945bb925e1648a35609c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } } }
I20260619 18:41:20.308926 9096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:20.309873 9102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:20.309687 9097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 681ac362c03945bb925e1648a35609c8. Latest consensus state: current_term: 1 leader_uuid: "681ac362c03945bb925e1648a35609c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "681ac362c03945bb925e1648a35609c8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35527 } } }
I20260619 18:41:20.310153 9097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:20.315665 9102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:20.325513 9102 catalog_manager.cc:1347] Generated new cluster ID: 57407aeaefc44979beb58bfd9f22ae0f
I20260619 18:41:20.325683 9102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:20.339309 9102 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:20.340996 9102 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:20.355263 9102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 681ac362c03945bb925e1648a35609c8: Generated new TSK 0
I20260619 18:41:20.356207 9102 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:41:20.410333 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:20.789969 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:20.790277 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:20.790432 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:20.798425 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:20.798660 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:20.798906 9116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:20.810231 9116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:20.811708 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:20.813257 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:20.828738 9122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:20.828938 9124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:20.828770 9121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:20.829623 9116 server_base.cc:1034] running on GCE node
I20260619 18:41:20.830327 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:20.831465 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:20.832702 9116 hybrid_clock.cc:648] HybridClock initialized: now 1781894480832644 us; error 46 us; skew 500 ppm
I20260619 18:41:20.833108 9116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:20 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:20.840426 9116 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:41:20.842125 9116 webserver.cc:460] Webserver started at http://127.3.85.129:45765/ using document root <none> and password file <none>
I20260619 18:41:20.842929 9116 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:20.843042 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:20.843320 9116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:20.846603 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "4323c225abd7490d8a911ae48e603ced"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "ff7ed0b5b925c3fa541507a4d5b20fc6"
server_key_iv: "f3310c85053269991fe8be9c6cfc27f2"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.847455 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "4323c225abd7490d8a911ae48e603ced"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "ff7ed0b5b925c3fa541507a4d5b20fc6"
server_key_iv: "f3310c85053269991fe8be9c6cfc27f2"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.853144 9116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:41:20.857010 9131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:20.858726 9116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:41:20.858882 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "4323c225abd7490d8a911ae48e603ced"
format_stamp: "Formatted at 2026-06-19 18:41:20 on dist-test-slave-09fp"
server_key: "ff7ed0b5b925c3fa541507a4d5b20fc6"
server_key_iv: "f3310c85053269991fe8be9c6cfc27f2"
server_key_version: "encryptionkey@0"
I20260619 18:41:20.859074 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:20.873525 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:20.878189 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:20.878566 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:20.879526 9116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:20.880872 9116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:20.880944 9116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:20.881031 9116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:20.881078 9116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:20.916417 9116 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:34791
I20260619 18:41:20.916438 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:34791 every 8 connection(s)
I20260619 18:41:20.917876 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:20.926430 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9116
I20260619 18:41:20.926677 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:20.927467 3414 external_mini_cluster.cc:1428] Setting key d554fa9f930fe9d07e3f2d8eff9825ec
Jun 19 18:41:20 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:20.940939 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.920757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:49021 (local address 127.3.85.190:35527)
0619 18:41:20.921251 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:20.921258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:20.922422 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 18:41:20.924354 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:20.924359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:20.924443 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:20.924545 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:20.926201 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.927425 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.929504 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.930107 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.930258 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:20.934120 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:20.934130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:20.934138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:20.934188 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:20.937095 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:20.938307 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:20.938315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:20.938318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:20.938525 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:20.939074 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:20.939077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:20.939080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:20.939361 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:20.939465 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:20.940458 (+ 993us) server_negotiation.cc:299] Negotiation successful
0619 18:41:20.940660 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260619 18:41:20.942620 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.921360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35527 (local address 127.3.85.129:49021)
0619 18:41:20.922230 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:20.922286 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:20.924037 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:20.924771 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:20.924780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:20.925355 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:20.925981 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:20.926000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.927698 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.927704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:20.929312 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:20.929331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:20.930378 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:20.930384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:20.930712 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:20.931481 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:20.931506 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:20.933815 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:20.937300 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:20.937314 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:20.937338 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:20.938054 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:20.938662 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:20.938670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:20.938676 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:20.938956 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:20.939507 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:20.939519 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:20.940019 (+ 500us) client_negotiation.cc:769] Sending connection context
0619 18:41:20.940341 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 18:41:20.940993 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":319,"threads_started":1}
I20260619 18:41:20.944412 9244 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35527
I20260619 18:41:20.944796 9244 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:20.945744 9244 heartbeater.cc:511] Master 127.3.85.190:35527 requested a full tablet report, sending...
I20260619 18:41:20.948287 9055 ts_manager.cc:194] Registered new tserver with Master: 4323c225abd7490d8a911ae48e603ced (127.3.85.129:34791)
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:20.950132 9055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:49021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:41:20.978421 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:21.321583 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:21.321969 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:21.322144 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:21.330081 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:21.330261 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:21.330477 9250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:21.340703 9250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:21.342108 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:21.343525 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:21.357560 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:21.357584 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:21.357673 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:21.358269 9250 server_base.cc:1034] running on GCE node
I20260619 18:41:21.359099 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:21.360131 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:21.361402 9250 hybrid_clock.cc:648] HybridClock initialized: now 1781894481361345 us; error 49 us; skew 500 ppm
I20260619 18:41:21.361771 9250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:21 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:21.365252 9250 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:41:21.367053 9250 webserver.cc:460] Webserver started at http://127.3.85.130:44693/ using document root <none> and password file <none>
I20260619 18:41:21.367864 9250 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:21.368032 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:21.368381 9250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:21.371762 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "9baabe97060c4097962a0d96a74e2090"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "7bf9d5506c14ab4c96d1f6d36e386ddc"
server_key_iv: "2e11018a6cdeaefff232dbc44509fd13"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.372713 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "9baabe97060c4097962a0d96a74e2090"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "7bf9d5506c14ab4c96d1f6d36e386ddc"
server_key_iv: "2e11018a6cdeaefff232dbc44509fd13"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.378597 9250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:41:21.382345 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.383924 9250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:21.384138 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "9baabe97060c4097962a0d96a74e2090"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "7bf9d5506c14ab4c96d1f6d36e386ddc"
server_key_iv: "2e11018a6cdeaefff232dbc44509fd13"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.384373 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:21.398026 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:21.402985 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:21.403319 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:21.404368 9250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:21.405846 9250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:21.405961 9250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.406123 9250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:21.406203 9250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.441581 9250 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:42211
I20260619 18:41:21.441677 9377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:42211 every 8 connection(s)
I20260619 18:41:21.443101 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:21.450886 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9250
I20260619 18:41:21.451062 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:21.451503 3414 external_mini_cluster.cc:1428] Setting key 51d3ff7a463e8166bcfbdcf9441247f6
Jun 19 18:41:21 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:21.464782 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.445876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44235 (local address 127.3.85.190:35527)
0619 18:41:21.446312 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:21.446317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:21.447441 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 18:41:21.449336 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:21.449341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:21.449451 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:21.449575 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:21.451369 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.452230 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.454110 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.454629 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.454771 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:21.458548 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:21.458561 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:21.458565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:21.458601 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:21.461316 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:21.462213 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:21.462220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:21.462225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:21.462393 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:21.462889 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:21.462895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:21.462900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:21.463241 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:21.463407 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:21.464148 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 18:41:21.464397 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260619 18:41:21.465853 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.446429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35527 (local address 127.3.85.130:44235)
0619 18:41:21.447241 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:21.447283 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:21.449007 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:21.449843 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:21.449852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:21.450394 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:21.451174 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:21.451202 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.452428 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.452436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:21.453938 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:21.453954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.454892 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.454900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:21.455151 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:21.455902 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:21.455927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:21.458184 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:21.461505 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:21.461516 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:21.461536 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:21.462079 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:21.462536 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:21.462541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:21.462546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:21.462758 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:21.463400 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:21.463409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:21.463781 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 18:41:21.464073 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 18:41:21.464660 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":289,"threads_started":1}
I20260619 18:41:21.467628 9378 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35527
I20260619 18:41:21.468029 9378 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:21.468843 9378 heartbeater.cc:511] Master 127.3.85.190:35527 requested a full tablet report, sending...
I20260619 18:41:21.470546 9055 ts_manager.cc:194] Registered new tserver with Master: 9baabe97060c4097962a0d96a74e2090 (127.3.85.130:42211)
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:41:21.471431 9055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:44235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:41:21.499696 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:21.846101 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:21.846412 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:21.846626 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:21.854486 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:21.854919 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:21.855122 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:21.865906 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:35527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:21.867363 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:21.868734 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:21.883477 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:21.883551 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:21.883843 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:21.884552 9384 server_base.cc:1034] running on GCE node
I20260619 18:41:21.885289 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:21.886319 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:21.887570 9384 hybrid_clock.cc:648] HybridClock initialized: now 1781894481887523 us; error 44 us; skew 500 ppm
I20260619 18:41:21.887928 9384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:21 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:21.891360 9384 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:41:21.893250 9384 webserver.cc:460] Webserver started at http://127.3.85.131:44135/ using document root <none> and password file <none>
I20260619 18:41:21.894038 9384 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:21.894199 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:21.894613 9384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:21.897730 9384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "f30faf3a471d4b0682b3587216d9934c"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "aacec5eb88acc4404ebd62ceeabe9a64"
server_key_iv: "e6a20ccb1fde7357619db28377ea6cd1"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.898710 9384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "f30faf3a471d4b0682b3587216d9934c"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "aacec5eb88acc4404ebd62ceeabe9a64"
server_key_iv: "e6a20ccb1fde7357619db28377ea6cd1"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.904253 9384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:21.908413 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.909883 9384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:41:21.910089 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "f30faf3a471d4b0682b3587216d9934c"
format_stamp: "Formatted at 2026-06-19 18:41:21 on dist-test-slave-09fp"
server_key: "aacec5eb88acc4404ebd62ceeabe9a64"
server_key_iv: "e6a20ccb1fde7357619db28377ea6cd1"
server_key_version: "encryptionkey@0"
I20260619 18:41:21.910346 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:21.929239 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:21.934078 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:21.934412 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:21.935515 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:21.937018 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:21.937131 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.937287 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:21.937363 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:21.954526 9244 heartbeater.cc:503] Master 127.3.85.190:35527 was elected leader, sending a full tablet report...
I20260619 18:41:21.973407 9384 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:43443
I20260619 18:41:21.973456 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:43443 every 8 connection(s)
I20260619 18:41:21.974874 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:21.982280 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9384
I20260619 18:41:21.982494 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:21.983012 3414 external_mini_cluster.cc:1428] Setting key 80e4efc1a286ee6a649748e4c094b04e
Jun 19 18:41:21 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:21.997715 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.977602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:46291 (local address 127.3.85.190:35527)
0619 18:41:21.978078 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:21.978084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:21.979358 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 18:41:21.981106 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:21.981111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:21.981199 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:21.981300 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:21.983451 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.984451 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.986506 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.986917 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.987061 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:21.991173 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:21.991179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:21.991183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:21.991231 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:21.993993 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:21.995049 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:21.995054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:21.995057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:21.995198 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:21.995684 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:21.995687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:21.995689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:21.995992 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:21.996117 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:21.997011 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 18:41:21.997299 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
I20260619 18:41:21.998968 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.978220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:35527 (local address 127.3.85.131:46291)
0619 18:41:21.979157 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:21.979206 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:21.980834 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:21.981532 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:21.981540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:21.982337 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:21.983218 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:21.983250 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.984634 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.984638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:21.986325 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:21.986341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:21.987180 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:21.987187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:21.987433 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:21.988202 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:21.988231 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:21.990887 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:21.994197 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:21.994208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:21.994225 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:21.994820 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:21.995310 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:21.995314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:21.995316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:21.995553 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:21.996132 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:21.996139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:21.996600 (+ 461us) client_negotiation.cc:769] Sending connection context
0619 18:41:21.996903 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 18:41:21.997449 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
I20260619 18:41:22.000387 9512 heartbeater.cc:347] Connected to a master server at 127.3.85.190:35527
I20260619 18:41:22.000710 9512 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:22.001502 9512 heartbeater.cc:511] Master 127.3.85.190:35527 requested a full tablet report, sending...
I20260619 18:41:22.003139 9055 ts_manager.cc:194] Registered new tserver with Master: f30faf3a471d4b0682b3587216d9934c (127.3.85.131:43443)
I20260619 18:41:22.003953 9055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:46291
I20260619 18:41:22.011677 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/test-admin.keytab.
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.040804 3414 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 18:41:22.046288 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:22.043508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.3.85.129:34791)
0619 18:41:22.044129 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.044141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.044236 (+ 95us) server_negotiation.cc:407] Connection header received
0619 18:41:22.044475 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.044483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.044703 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.044893 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.045832 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34568: BlockingRecv error: recv got EOF from 127.0.0.1:34568 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
W20260619 18:41:22.046432 3414 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.048185 3414 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.049598 3414 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:22.066550 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.051595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.3.85.129:34791)
0619 18:41:22.051841 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.051849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.052068 (+ 219us) server_negotiation.cc:407] Connection header received
0619 18:41:22.052227 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.052230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.052296 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.052393 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.053432 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.054529 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.055808 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.056416 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.056525 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.060494 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.060509 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.060526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.060575 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.063744 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.064585 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.064593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.064617 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.064769 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.065121 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.065125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.065127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.065500 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.065739 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.066009 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.066223 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 18:41:22.072855 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:22.070893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.3.85.129:34791)
0619 18:41:22.071077 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.071081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.071341 (+ 260us) server_negotiation.cc:407] Connection header received
0619 18:41:22.071522 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.071527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.071609 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.071702 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.072558 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34592: BlockingRecv error: recv got EOF from 127.0.0.1:34592 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 18:41:22.073091 3414 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.074769 3414 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.076150 3414 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:22.090977 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.077878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.3.85.129:34791)
0619 18:41:22.078078 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.078085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.078205 (+ 120us) server_negotiation.cc:407] Connection header received
0619 18:41:22.078346 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.078350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.078416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.078551 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.079544 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.080244 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.081455 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.081870 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.081965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.085779 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.085785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.085789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.085838 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.088362 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.089215 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.089219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.089221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.089349 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.089690 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.089694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.089696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.089955 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.090142 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.090376 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.090665 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 18:41:22.096035 9520 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:22.093732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.3.85.129:34791)
0619 18:41:22.093997 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.094006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.094356 (+ 350us) server_negotiation.cc:407] Connection header received
0619 18:41:22.094603 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.094607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.094689 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.094808 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.095736 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34606: BlockingRecv error: recv got EOF from 127.0.0.1:34606 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260619 18:41:22.096171 3414 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894429649633-3414-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.097885 3414 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:22.099367 3414 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:22.115232 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.100960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.3.85.129:34791)
0619 18:41:22.101210 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.101218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.101348 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:41:22.101505 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.101509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.101572 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.101657 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.102870 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.103641 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.104932 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.105338 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.105440 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.109610 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.109616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.109620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.109661 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.112501 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.113194 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.113198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.113201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.113409 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.113765 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.113768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.113770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.114056 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.114337 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.114630 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.114817 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 18:41:22.117256 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9116
I20260619 18:41:22.131551 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.118241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.3.85.129:34791)
0619 18:41:22.118503 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.118513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.118823 (+ 310us) server_negotiation.cc:407] Connection header received
0619 18:41:22.119048 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.119055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.119131 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.119248 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.120344 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.121073 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.122436 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.123030 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.123196 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.125790 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.125796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.125800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.125842 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.128508 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.129321 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.129328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.129332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.129498 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.129937 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.129950 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.129953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.130252 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.130595 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.130950 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.131196 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:41:22.132318 9239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:22.218073 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9116
I20260619 18:41:22.234828 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9250
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:41:22.251121 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.235703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.3.85.130:42211)
0619 18:41:22.236258 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.236265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.236291 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:22.236351 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.236358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.236571 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.236754 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.237737 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.238823 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.240056 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.240642 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.240800 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.244892 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.244908 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.244926 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.244968 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.247887 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.248776 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.248786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.248788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.248926 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.249330 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.249336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.249341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.249714 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.250055 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.250394 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.250820 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
I20260619 18:41:22.252491 9373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:22.335326 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9250
I20260619 18:41:22.351172 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9384
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:41:22.367822 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.352093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.3.85.131:43443)
0619 18:41:22.352668 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:22.352675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:22.352704 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:41:22.352776 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:22.352783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:22.352996 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:22.353181 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:22.354254 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.355350 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.356670 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:22.357268 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:22.357419 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:22.361601 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:22.361617 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:22.361635 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:22.361677 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:22.365054 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.365856 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.365863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.365865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.366001 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:22.366358 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:22.366361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:22.366363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:22.366811 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:22.367060 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:22.367307 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:41:22.367520 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":268,"threads_started":1}
I20260619 18:41:22.369120 9507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:22.454119 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9384
I20260619 18:41:22.470099 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9023
I20260619 18:41:22.471379 9085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:22.573283 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9023
2026-06-19T18:41:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4119 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9540](info): set up 2 sockets
Jun 19 18:41:22 dist-test-slave-09fp krb5kdc[9540](info): commencing operation
krb5kdc: starting...
W20260619 18:41:23.675755 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:23 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab.
Entry for principal oryx/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab.
I20260619 18:41:23.903543 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:45005
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab
--principal=oryx/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:24.243443 9553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:24.243773 9553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:24.243948 9553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:24.251748 9553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:24.251899 9553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:24.252007 9553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:24.252099 9553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:24.252182 9553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:24.262534 9553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab
--principal=oryx/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:24.263991 9553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:24.265357 9553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:24.278963 9558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:24.279021 9561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:24.279052 9559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:24.284356 9553 server_base.cc:1034] running on GCE node
I20260619 18:41:24.285274 9553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:24.286913 9553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:24.288183 9553 hybrid_clock.cc:648] HybridClock initialized: now 1781894484288135 us; error 62 us; skew 500 ppm
I20260619 18:41:24.288585 9553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:24 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894484, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:24.291867 9553 init.cc:377] Logged in from keytab as oryx/127.3.85.190@KRBTEST.COM (short username oryx)
I20260619 18:41:24.293457 9553 webserver.cc:460] Webserver started at http://127.3.85.190:33859/ using document root <none> and password file <none>
I20260619 18:41:24.294185 9553 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:24.294292 9553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:24.294659 9553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:24.297806 9553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "7e6260ca280f49e888a6fdd9ad2024a9"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "f207306489a2ccad3b76eb6112a517d3"
server_key_iv: "17e53aa3d4a84a273d8544125e8e4df2"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.298880 9553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "7e6260ca280f49e888a6fdd9ad2024a9"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "f207306489a2ccad3b76eb6112a517d3"
server_key_iv: "17e53aa3d4a84a273d8544125e8e4df2"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.304338 9553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:24.308077 9568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:24.309479 9553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:41:24.309687 9553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "7e6260ca280f49e888a6fdd9ad2024a9"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "f207306489a2ccad3b76eb6112a517d3"
server_key_iv: "17e53aa3d4a84a273d8544125e8e4df2"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.309901 9553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:24.335157 9553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:24.339434 9553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:24.339766 9553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:24.359359 9553 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:45005
I20260619 18:41:24.359390 9619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:45005 every 8 connection(s)
I20260619 18:41:24.360842 9553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:24.365499 9620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:24.366209 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9553
I20260619 18:41:24.366420 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:24.367108 3414 external_mini_cluster.cc:1428] Setting key d82d1a4ea388e687115cc14b388f3df9
I20260619 18:41:24.381126 9620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Bootstrap starting.
Jun 19 18:41:24 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:24.384711 9620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:24.386217 9620 log.cc:824] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:24.390224 9620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: No bootstrap required, opened a new log
I20260619 18:41:24.394304 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:24.369344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.3.85.190:45005)
0619 18:41:24.370237 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:24.370259 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:24.370318 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:41:24.372464 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:24.372501 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:24.373133 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:24.373653 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:24.375242 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:24.377445 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:24.378946 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:24.380141 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:24.380366 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:24.384726 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:24.384742 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:24.384766 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:24.384821 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:24.389016 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:24.389902 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:24.389912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:24.389920 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:24.390139 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:24.390704 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:24.390709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:24.390712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:24.391502 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:24.391861 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:24.392197 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:41:24.392737 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":365,"threads_started":1}
I20260619 18:41:24.400408 9620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } }
I20260619 18:41:24.400835 9620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:24.400903 9620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6260ca280f49e888a6fdd9ad2024a9, State: Initialized, Role: FOLLOWER
I20260619 18:41:24.401551 9620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } }
I20260619 18:41:24.401785 9620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:24.401880 9620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:24.402007 9620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:24.404690 9620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } }
I20260619 18:41:24.405226 9620 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6260ca280f49e888a6fdd9ad2024a9; no voters:
I20260619 18:41:24.405846 9620 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:24.405959 9625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:24.406810 9625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6260ca280f49e888a6fdd9ad2024a9, State: Running, Role: LEADER
I20260619 18:41:24.407511 9625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } }
I20260619 18:41:24.408537 9620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:24.410847 9627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6260ca280f49e888a6fdd9ad2024a9. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } }
I20260619 18:41:24.411339 9627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:24.411936 9626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } }
I20260619 18:41:24.412308 9626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:24.414521 9631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:24.420975 9631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:24.430994 9631 catalog_manager.cc:1347] Generated new cluster ID: 6bf7eb60acdf4907a37fbb3e89abf98f
I20260619 18:41:24.431136 9631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:24.448722 9631 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:24.450699 9631 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:24.461017 9631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Generated new TSK 0
I20260619 18:41:24.462312 9631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
I20260619 18:41:24.519762 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:24.876461 9646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:24.876775 9646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:24.876869 9646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:24.884503 9646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:24.884634 9646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:24.884747 9646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:24.895041 9646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:24.896464 9646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:24.897836 9646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:24.912037 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:24.911980 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:24.911980 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:24.912817 9646 server_base.cc:1034] running on GCE node
I20260619 18:41:24.913570 9646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:24.914834 9646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:24.916062 9646 hybrid_clock.cc:648] HybridClock initialized: now 1781894484916013 us; error 48 us; skew 500 ppm
I20260619 18:41:24.916414 9646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:24 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894484, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:24.920092 9646 init.cc:377] Logged in from keytab as oryx/127.3.85.129@KRBTEST.COM (short username oryx)
I20260619 18:41:24.921752 9646 webserver.cc:460] Webserver started at http://127.3.85.129:41785/ using document root <none> and password file <none>
I20260619 18:41:24.922580 9646 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:24.922730 9646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:24.923043 9646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:24.926216 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "fcaf9febf06f4bafa622d2bd2b53f49a"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "e1883a6a8930d91afc03b797a813b6e4"
server_key_iv: "5253aac5586b81b967bb7c2d447ff91f"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.927163 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "fcaf9febf06f4bafa622d2bd2b53f49a"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "e1883a6a8930d91afc03b797a813b6e4"
server_key_iv: "5253aac5586b81b967bb7c2d447ff91f"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.933048 9646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:24.936964 9661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:24.938592 9646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:24.938797 9646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "fcaf9febf06f4bafa622d2bd2b53f49a"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "e1883a6a8930d91afc03b797a813b6e4"
server_key_iv: "5253aac5586b81b967bb7c2d447ff91f"
server_key_version: "encryptionkey@0"
I20260619 18:41:24.939030 9646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:24.960655 9646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:24.965507 9646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:24.965837 9646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:24.966930 9646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:24.968333 9646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:24.968410 9646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:24.968547 9646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:24.968606 9646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:25.001206 9646 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:40031
I20260619 18:41:25.001266 9773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:40031 every 8 connection(s)
I20260619 18:41:25.002574 9646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:25.005129 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9646
I20260619 18:41:25.005345 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:25.006173 3414 external_mini_cluster.cc:1428] Setting key cba21040a31af330d6299dbd82399cce
Jun 19 18:41:25 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894484, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:25.025695 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.005489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:35443 (local address 127.3.85.190:45005)
0619 18:41:25.005988 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:25.005992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:25.007293 (+ 1301us) server_negotiation.cc:407] Connection header received
0619 18:41:25.009639 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:25.009645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:25.009735 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:25.009872 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:25.011604 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.012336 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.014224 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.014871 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.014994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:25.019616 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:25.019621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:25.019625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:25.019659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:25.022297 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:25.023303 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:25.023309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:25.023312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:25.023449 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:25.023914 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:25.023924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:25.023926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:25.024187 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:25.024352 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:25.025086 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 18:41:25.025317 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for oryx/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:25.026793 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.006180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:35443)
0619 18:41:25.007115 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:25.007168 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:25.009315 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:25.010155 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:25.010166 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:25.010802 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:25.011404 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:25.011422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.012511 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.012516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:25.014058 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:25.014077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.015135 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.015145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:25.015491 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:25.016276 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:25.016312 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:25.019338 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:25.022502 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:25.022513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:25.022534 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:25.023152 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:25.023579 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:25.023584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:25.023588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:25.023785 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:25.024324 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:25.024332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:25.024707 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 18:41:25.024986 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 18:41:25.025548 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":379,"threads_started":1}
I20260619 18:41:25.028417 9774 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:25.028712 9774 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.130@KRBTEST.COM" created.
I20260619 18:41:25.029495 9774 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:25.031771 9585 ts_manager.cc:194] Registered new tserver with Master: fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:25.033326 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:35443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
I20260619 18:41:25.053808 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:25.394309 9780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:25.394711 9780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:25.394811 9780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:25.402976 9780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:25.403123 9780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:25.403240 9780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:25.413556 9780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:25.415107 9780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:25.416513 9780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:25.431396 9785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:25.431348 9786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:25.431363 9788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:25.432408 9780 server_base.cc:1034] running on GCE node
I20260619 18:41:25.433187 9780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:25.434387 9780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:25.435655 9780 hybrid_clock.cc:648] HybridClock initialized: now 1781894485435582 us; error 71 us; skew 500 ppm
I20260619 18:41:25.435986 9780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:25 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894485, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:25.439440 9780 init.cc:377] Logged in from keytab as oryx/127.3.85.130@KRBTEST.COM (short username oryx)
I20260619 18:41:25.441032 9780 webserver.cc:460] Webserver started at http://127.3.85.130:34341/ using document root <none> and password file <none>
I20260619 18:41:25.441782 9780 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:25.441921 9780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:25.442253 9780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:25.445569 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "b06c661541a74280b3a14c8c52242609"
format_stamp: "Formatted at 2026-06-19 18:41:25 on dist-test-slave-09fp"
server_key: "82ae468f88ce2594f0b43daab41cf75f"
server_key_iv: "adae3e4519ef08a92baf8f4b0376bc57"
server_key_version: "encryptionkey@0"
I20260619 18:41:25.446422 9780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "b06c661541a74280b3a14c8c52242609"
format_stamp: "Formatted at 2026-06-19 18:41:25 on dist-test-slave-09fp"
server_key: "82ae468f88ce2594f0b43daab41cf75f"
server_key_iv: "adae3e4519ef08a92baf8f4b0376bc57"
server_key_version: "encryptionkey@0"
I20260619 18:41:25.452066 9780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:25.455968 9795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:25.457574 9780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:25.457787 9780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "b06c661541a74280b3a14c8c52242609"
format_stamp: "Formatted at 2026-06-19 18:41:25 on dist-test-slave-09fp"
server_key: "82ae468f88ce2594f0b43daab41cf75f"
server_key_iv: "adae3e4519ef08a92baf8f4b0376bc57"
server_key_version: "encryptionkey@0"
I20260619 18:41:25.458010 9780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:25.501811 9780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:25.506601 9780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:25.506942 9780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:25.508001 9780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:25.509511 9780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:25.509598 9780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:25.509773 9780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:25.509822 9780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:25.545593 9780 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:45429
I20260619 18:41:25.545696 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:45429 every 8 connection(s)
I20260619 18:41:25.547052 9780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:25.556685 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9780
I20260619 18:41:25.556892 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:25.557360 3414 external_mini_cluster.cc:1428] Setting key a8846ca5a2e40fbeda9e17809e36dd75
Jun 19 18:41:25 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894485, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:25.571049 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.549766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:36883 (local address 127.3.85.190:45005)
0619 18:41:25.550243 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:25.550248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:25.551427 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 18:41:25.553279 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:25.553283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:25.553356 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:25.553468 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:25.555289 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.556041 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.557884 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.558663 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.558772 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:25.563437 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:25.563448 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:25.563456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:25.563507 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:25.567100 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:25.568335 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:25.568340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:25.568342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:25.568486 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:25.569030 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:25.569033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:25.569035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:25.569344 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:25.569461 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:25.570564 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0619 18:41:25.570762 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260619 18:41:25.572651 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.550388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:36883)
0619 18:41:25.551234 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:25.551279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:25.552972 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:25.553749 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:25.553758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:25.554354 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:25.555100 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:25.555118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.556199 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.556204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:25.557706 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:25.557721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:25.558929 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:25.558938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:25.559248 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:25.560211 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:25.560236 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:25.563106 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:25.567332 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:25.567346 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:25.567373 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:25.568158 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:25.568609 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:25.568617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:25.568624 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:25.568900 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:25.569483 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:25.569495 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:25.570021 (+ 526us) client_negotiation.cc:769] Sending connection context
0619 18:41:25.570363 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 18:41:25.571041 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":291,"threads_started":1}
I20260619 18:41:25.574592 9908 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:25.574985 9908 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:25.575939 9908 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:25.577993 9585 ts_manager.cc:194] Registered new tserver with Master: b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429)
I20260619 18:41:25.579412 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:36883
WARNING: no policy specified for oryx/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
I20260619 18:41:25.610910 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:25.955847 9914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:25.956156 9914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:25.956315 9914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:25.964063 9914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:25.964195 9914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:25.964370 9914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:25.974610 9914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:25.975996 9914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:25.977380 9914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:25.991416 9919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:25.991583 9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:25.991441 9920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:25.992342 9914 server_base.cc:1034] running on GCE node
I20260619 18:41:25.993078 9914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:25.994189 9914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:25.995431 9914 hybrid_clock.cc:648] HybridClock initialized: now 1781894485995372 us; error 73 us; skew 500 ppm
I20260619 18:41:25.995769 9914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:25 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894485, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:26.002737 9914 init.cc:377] Logged in from keytab as oryx/127.3.85.131@KRBTEST.COM (short username oryx)
I20260619 18:41:26.004434 9914 webserver.cc:460] Webserver started at http://127.3.85.131:45427/ using document root <none> and password file <none>
I20260619 18:41:26.005254 9914 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:26.005395 9914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:26.005725 9914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:26.008949 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "88d46ba6f5064d47873f82f37d80a241"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "64250a675149da835d66bc6a453917f9"
server_key_iv: "761143b92b8e229a6a7495432246bd87"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.009791 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "88d46ba6f5064d47873f82f37d80a241"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "64250a675149da835d66bc6a453917f9"
server_key_iv: "761143b92b8e229a6a7495432246bd87"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.015745 9914 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 18:41:26.019744 9929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.021268 9914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:26.021456 9914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "88d46ba6f5064d47873f82f37d80a241"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "64250a675149da835d66bc6a453917f9"
server_key_iv: "761143b92b8e229a6a7495432246bd87"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.021660 9914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:26.036700 9774 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
I20260619 18:41:26.048424 9914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:26.052917 9914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:26.053189 9914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:26.054145 9914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:26.055665 9914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:26.055744 9914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.055832 9914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:26.055877 9914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.092414 9914 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46161
I20260619 18:41:26.092468 10041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46161 every 8 connection(s)
I20260619 18:41:26.093858 9914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:26.103854 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 9914
I20260619 18:41:26.104066 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:26.104519 3414 external_mini_cluster.cc:1428] Setting key 4e0f204d7b63f0a9774c96406f133dd3
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894485, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:26.116370 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.097008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:50455 (local address 127.3.85.190:45005)
0619 18:41:26.097502 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.097507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.098689 (+ 1182us) server_negotiation.cc:407] Connection header received
0619 18:41:26.100589 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.100594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.100694 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.100819 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.102474 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.103271 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.105111 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.105653 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.105813 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.110064 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.110070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.110073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.110109 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.112885 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.113776 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.113782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.113792 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.113955 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.114433 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.114439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.114482 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.114797 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.115004 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.115720 (+ 716us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.115965 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":232,"threads_started":1}
I20260619 18:41:26.117353 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.097643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:50455)
0619 18:41:26.098438 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.098562 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.100252 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.101084 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.101094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.101658 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.102252 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.102270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.103456 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.103460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.104917 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.104937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.105951 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.105960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.106196 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.107053 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.107085 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.109818 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.113059 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.113069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.113089 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.113648 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.114071 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.114077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.114081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.114291 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.114956 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.114965 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.115350 (+ 385us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.115640 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.116207 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":329,"threads_started":1}
I20260619 18:41:26.118958 10042 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:26.119333 10042 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:26.120311 10042 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:26.121822 9585 ts_manager.cc:194] Registered new tserver with Master: 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:26.122682 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:50455
I20260619 18:41:26.132997 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.85.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.189.keytab.
Entry for principal oryx/127.3.85.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.189.keytab.
I20260619 18:41:26.185643 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:45025
--webserver_interface=127.3.85.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.189.keytab
--principal=oryx/127.3.85.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:45005,127.3.85.189:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:26.529661 10048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:26.530055 10048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:26.530221 10048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:26.537891 10048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:26.538072 10048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:26.538184 10048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:26.538300 10048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:26.538388 10048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:26.548936 10048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:45005,127.3.85.189:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.189.keytab
--principal=oryx/127.3.85.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.85.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
W20260619 18:41:26.549397 10048 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.85.190:45005,127.3.85.189:45025'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 18:41:26.550557 10048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:26.551880 10048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:26.564157 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:26.564257 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:26.564239 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:26.565564 10048 server_base.cc:1034] running on GCE node
I20260619 18:41:26.566535 10048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:26.568092 10048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:26.569396 10048 hybrid_clock.cc:648] HybridClock initialized: now 1781894486569330 us; error 46 us; skew 500 ppm
I20260619 18:41:26.569796 10048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:26.573428 10048 init.cc:377] Logged in from keytab as oryx/127.3.85.189@KRBTEST.COM (short username oryx)
I20260619 18:41:26.575297 10048 webserver.cc:460] Webserver started at http://127.3.85.189:37071/ using document root <none> and password file <none>
I20260619 18:41:26.576086 10048 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:26.576228 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:26.576556 10048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:26.579779 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/instance:
uuid: "ddf9cd1c58a742ae918a390de219683e"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "ac24724b4a90c58ebca283c90be372ae"
server_key_iv: "59db844b91264dabbeab3bf7f6822994"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.580646 10048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal/instance:
uuid: "ddf9cd1c58a742ae918a390de219683e"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "ac24724b4a90c58ebca283c90be372ae"
server_key_iv: "59db844b91264dabbeab3bf7f6822994"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.583298 9908 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
I20260619 18:41:26.586885 10048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:41:26.590639 10063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.592121 10048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:26.592327 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "ddf9cd1c58a742ae918a390de219683e"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "ac24724b4a90c58ebca283c90be372ae"
server_key_iv: "59db844b91264dabbeab3bf7f6822994"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.592581 10048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:26.613370 10048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:26.619115 10048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:26.619354 10048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:26.638768 10048 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.189:45025
I20260619 18:41:26.638792 10114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.189:45025 every 8 connection(s)
I20260619 18:41:26.640206 10048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
I20260619 18:41:26.644304 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:26.647449 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10048
I20260619 18:41:26.651808 10115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.189@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:26.670879 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.655226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.3.85.190:45005)
0619 18:41:26.656688 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.656698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.656720 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:26.656782 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.656787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.656874 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.657003 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.658238 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.659079 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.660558 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.660973 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.661137 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.664913 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.664918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.664922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.664961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.667903 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.668705 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.668711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.668714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.668857 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.669238 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.669243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.669247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.669563 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.669735 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.670261 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.670554 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":1231,"threads_started":1}
I20260619 18:41:26.671998 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.655192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47654)
0619 18:41:26.656184 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.656253 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.656467 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.657075 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.657080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.657783 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.658049 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.658058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.659230 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.659235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.660369 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.660410 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.661240 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.661249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.661415 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.662092 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.662126 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.664673 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.668068 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.668082 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.668096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.668568 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.668973 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.668979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.668983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.669127 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.669698 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.669707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.669914 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.670172 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.670919 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":241,"spinlock_wait_cycles":19328,"thread_start_us":388,"threads_started":1}
I20260619 18:41:26.672017 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.649873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.3.85.189:45025)
0619 18:41:26.650783 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.650796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.650838 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:41:26.652532 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.652559 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.653039 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.653598 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.655204 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.657290 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.658657 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.659369 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.659623 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.664482 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.664493 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.664508 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.664544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.667402 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.668212 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.668219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.668221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.668400 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.668829 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.668834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.668839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.669403 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.669740 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.670156 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.670762 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":682,"thread_start_us":412,"threads_started":1}
I20260619 18:41:26.674983 10115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.189@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
I20260619 18:41:26.687950 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.675760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44578)
0619 18:41:26.675917 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.675946 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.676081 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.676370 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.676374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.676757 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.676884 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.676892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.678064 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.678069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.678991 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.679003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.679689 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.679694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.679940 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.680506 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.680523 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.682677 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.685706 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.685710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.685713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.686104 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.686437 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.686440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.686522 (+ 82us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.686659 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.687209 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.687212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.687349 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.687471 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.687690 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":44}
I20260619 18:41:26.688194 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.675941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.3.85.189:45025)
0619 18:41:26.676081 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.676087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.676104 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:41:26.676206 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.676209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.676258 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.676357 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.677030 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.677920 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.679134 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.679539 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.679679 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.682885 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.682891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.682894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.682926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.685547 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.686216 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.686220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.686223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.686334 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.686803 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.686808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.686810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.687103 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.687242 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.687684 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.687875 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":55}
I20260619 18:41:26.693243 10115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Bootstrap starting.
I20260619 18:41:26.696596 10115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:26.697734 10115 log.cc:824] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:26.700718 10115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: No bootstrap required, opened a new log
I20260619 18:41:26.711407 10115 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } }
I20260619 18:41:26.712016 10115 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:26.712170 10115 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf9cd1c58a742ae918a390de219683e, State: Initialized, Role: FOLLOWER
I20260619 18:41:26.713035 10115 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } }
I20260619 18:41:26.714138 10122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } } }
I20260619 18:41:26.714646 10122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:41:26.715742 10115 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:41:26.725950 10133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:26.726130 10133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:26.728542 10048 master_runner.cc:428] Detected that this master ddf9cd1c58a742ae918a390de219683e is joining an existing cluster
I20260619 18:41:26.728672 10048 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.85.189:45025
I20260619 18:41:26.750661 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.733515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.3.85.190:45005)
0619 18:41:26.733717 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.733734 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.734406 (+ 672us) server_negotiation.cc:407] Connection header received
0619 18:41:26.734765 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.734773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.734880 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.734959 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.736825 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.737903 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.739977 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.740626 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.740755 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.743239 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.743249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.743255 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.743301 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.747062 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.748514 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.748521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.748524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.748673 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.749086 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.749091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.749095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.749410 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.749569 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.750007 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.750269 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:41:26.750774 10139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.733538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47670)
0619 18:41:26.734277 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.734307 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.734578 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.735210 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.735217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.736081 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.736591 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.736609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.738071 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.738075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.739793 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.739811 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.740825 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.740831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.741024 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.741747 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.741778 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.743002 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.747886 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.747893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.747898 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.748369 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.748815 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.748819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.748821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.748960 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.749543 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.749548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.749741 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.749897 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.750268 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":1540,"threads_started":1}
I20260619 18:41:26.753749 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.734270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44582)
0619 18:41:26.738984 (+ 4714us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.739009 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.739142 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.739495 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.739500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.740146 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.740364 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.740372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.741712 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.741717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.743319 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.743329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.744477 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.744481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.744603 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.745168 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.745186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.746099 (+ 913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.749973 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.749977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.749981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.750626 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.751544 (+ 918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.751549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.751552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.751706 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.752527 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.752533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.752701 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.752855 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.753227 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4570,"thread_start_us":4355,"threads_started":1}
I20260619 18:41:26.754503 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.734859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.3.85.189:45025)
0619 18:41:26.735075 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.735081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.739113 (+ 4032us) server_negotiation.cc:407] Connection header received
0619 18:41:26.739256 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.739261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.739340 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.739448 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.740502 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.741495 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.743444 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.744067 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.744217 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.746295 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.746301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.746306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.746346 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.749815 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.751232 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.751238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.751241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.751398 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.751970 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.751975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.751978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.752384 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.752551 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.753004 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.753423 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":100}
WARNING: no policy specified for oryx/127.3.85.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.188@KRBTEST.COM" created.
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894484, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.190@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
I20260619 18:41:26.779551 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.761677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44592)
0619 18:41:26.762422 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.762521 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.762776 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.763593 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.763602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.764534 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.764869 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.764887 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.766255 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.766263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.767423 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.767439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.768343 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.768351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.768576 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.769319 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.769338 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.771676 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.775707 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.775728 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.775739 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.776442 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.777076 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.777085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.777091 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.777279 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.778037 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.778049 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.778311 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.778560 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.778995 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":576,"threads_started":1}
I20260619 18:41:26.780076 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.762069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.3.85.189:45025)
0619 18:41:26.762255 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.762261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.762611 (+ 350us) server_negotiation.cc:407] Connection header received
0619 18:41:26.763061 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.763067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.763188 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.763266 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.765074 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.766099 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.767612 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.768147 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.768300 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.771924 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.771930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.771933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.771969 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.775479 (+ 3510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.776669 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.776676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.776680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.776904 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.777446 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.777454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.777460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.777857 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.779275 (+ 1418us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.779466 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.779714 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:41:26.782485 9585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.85.189:45025: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ddf9cd1c58a742ae918a390de219683e"
member_type: NON_VOTER
last_known_addr {
host: "127.3.85.189"
port: 45025
}
attrs {
promote: true
}
}
dest_uuid: "7e6260ca280f49e888a6fdd9ad2024a9"
cas_config_opid_index: -1
I20260619 18:41:26.783811 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } }
I20260619 18:41:26.794106 10146 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ddf9cd1c58a742ae918a390de219683e (127.3.85.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } }
I20260619 18:41:26.795336 10090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:26.799683 10146 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.85.189:45025
I20260619 18:41:26.799993 10090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 18:41:26.800231 10146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } }
I20260619 18:41:26.800448 10146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:26.800571 10048 master.cc:537] Master@127.3.85.189:45025 shutting down...
I20260619 18:41:26.801689 10145 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.188.keytab.
Entry for principal oryx/127.3.85.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.188.keytab.
I20260619 18:41:26.806022 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:33659
--webserver_interface=127.3.85.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.188.keytab
--principal=oryx/127.3.85.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:26.806870 9569 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 18:41:26.809346 9569 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 -> Peer ddf9cd1c58a742ae918a390de219683e (127.3.85.189:45025): Couldn't send request to peer ddf9cd1c58a742ae918a390de219683e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:41:26.814401 10048 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:26.815141 10048 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:26.815346 10048 tablet_replica.cc:330] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: stopping tablet replica
I20260619 18:41:26.822331 10048 master.cc:552] Master@127.3.85.189:45025 shutdown complete.
I20260619 18:41:26.822412 10048 master_runner.cc:305] Clearing existing system tablet
I20260619 18:41:26.822865 10048 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:26.822999 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:26.825301 10048 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:41:26.830880 10151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.831276 10048 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:26.831564 10048 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260619 18:41:26.831661 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "ddf9cd1c58a742ae918a390de219683e"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "ac24724b4a90c58ebca283c90be372ae"
server_key_iv: "59db844b91264dabbeab3bf7f6822994"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.835249 10048 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:26.840580 10048 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:41:26.840803 10048 log.cc:1196] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 18:41:26.841271 10048 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Deleting consensus metadata
I20260619 18:41:26.841549 10048 master_runner.cc:315] Copying system tablet from 127.3.85.190:45005
I20260619 18:41:26.844142 10048 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.85.190:45005
I20260619 18:41:26.857221 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.845208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.3.85.190:45005)
0619 18:41:26.845437 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:26.845444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:26.845769 (+ 325us) server_negotiation.cc:407] Connection header received
0619 18:41:26.846096 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:26.846101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:26.846162 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:26.846274 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:26.847349 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.848074 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.849337 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.849737 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.849874 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.851749 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:26.851757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:26.851767 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:26.851799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:26.854440 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.855182 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.855189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.855192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.855330 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:26.855702 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:26.855708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:26.855711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:26.855994 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:26.856158 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:26.856605 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 18:41:26.856828 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:41:26.857574 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.845152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47686)
0619 18:41:26.845661 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.845677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:26.845850 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:26.846573 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:26.846578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:26.846950 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:26.847188 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.847196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.848213 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.848220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.849176 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:26.849192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:26.849889 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:26.849895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:26.850068 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:26.850792 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:26.850818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:26.851533 (+ 715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:26.854656 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.854662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.854666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.855052 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.855445 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:26.855450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:26.855454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:26.855592 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:26.856219 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:26.856224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:26.856367 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 18:41:26.856519 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 18:41:26.856834 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":276,"threads_started":1}
I20260619 18:41:26.858384 9605 tablet_copy_service.cc:140] P 7e6260ca280f49e888a6fdd9ad2024a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ddf9cd1c58a742ae918a390de219683e ({username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:47686)
I20260619 18:41:26.858702 9605 tablet_copy_service.cc:161] P 7e6260ca280f49e888a6fdd9ad2024a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ddf9cd1c58a742ae918a390de219683e at {username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:47686: session id = ddf9cd1c58a742ae918a390de219683e-00000000000000000000000000000000
I20260619 18:41:26.861735 9605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:41:26.863801 10048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:41:26.868674 10048 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: tablet copy: Starting download of 0 data blocks...
I20260619 18:41:26.869020 10048 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: tablet copy: Starting download of 1 WAL segments...
I20260619 18:41:26.871661 10048 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:41:26.875159 9605 tablet_copy_service.cc:342] P 7e6260ca280f49e888a6fdd9ad2024a9: Request end of tablet copy session ddf9cd1c58a742ae918a390de219683e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:47686
I20260619 18:41:26.875404 9605 tablet_copy_service.cc:433] P 7e6260ca280f49e888a6fdd9ad2024a9: ending tablet copy session ddf9cd1c58a742ae918a390de219683e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ddf9cd1c58a742ae918a390de219683e
I20260619 18:41:26.885805 10048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:26.888842 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:26.894847 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:26.902324 10048 server_base.cc:1034] running on GCE node
W20260619 18:41:26.902791 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:26.903230 10048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:26.903918 10048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:26.905139 10048 hybrid_clock.cc:648] HybridClock initialized: now 1781894486905076 us; error 37 us; skew 500 ppm
I20260619 18:41:26.905375 10048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:26 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894486, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:26.907428 10048 init.cc:377] Logged in from keytab as oryx/127.3.85.189@KRBTEST.COM (short username oryx)
I20260619 18:41:26.911401 10048 webserver.cc:460] Webserver started at http://127.3.85.189:43327/ using document root <none> and password file <none>
I20260619 18:41:26.911743 10048 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:26.911836 10048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:26.914533 10048 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:41:26.916404 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:26.916862 10048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 18:41:26.917018 10048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "ddf9cd1c58a742ae918a390de219683e"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "ac24724b4a90c58ebca283c90be372ae"
server_key_iv: "59db844b91264dabbeab3bf7f6822994"
server_key_version: "encryptionkey@0"
I20260619 18:41:26.917343 10048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:26.930678 10048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:26.934194 10048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:26.960692 10048 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.189:45025
I20260619 18:41:26.960728 10219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.189:45025 every 8 connection(s)
I20260619 18:41:26.961627 10048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
I20260619 18:41:26.969241 10220 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:41:26.974912 10220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Bootstrap starting.
I20260619 18:41:26.988727 10220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:41:26.989212 10220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Bootstrap complete.
I20260619 18:41:26.990096 10220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } }
I20260619 18:41:26.990341 10220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ddf9cd1c58a742ae918a390de219683e, State: Initialized, Role: LEARNER
I20260619 18:41:26.990648 10220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } }
I20260619 18:41:26.991526 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } }
I20260619 18:41:26.991851 10220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:26.991874 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:41:27.000089 10234 catalog_manager.cc:1259] Loaded cluster ID: 6bf7eb60acdf4907a37fbb3e89abf98f
I20260619 18:41:27.000324 10234 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: loading cluster ID for follower catalog manager: success
I20260619 18:41:27.003976 10234 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: acquiring CA information for follower catalog manager: success
I20260619 18:41:27.006207 10234 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:41:27.125411 10042 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
W20260619 18:41:27.236411 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:27.236729 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:27.236910 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:27.244619 10149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:27.244741 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:27.244787 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:27.244844 10149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:27.244894 10149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:27.255577 10149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.188.keytab
--principal=oryx/127.3.85.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.85.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:27.257020 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:27.258379 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:27.271871 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:27.271834 10242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:27.271781 10240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:27.273211 10149 server_base.cc:1034] running on GCE node
I20260619 18:41:27.274085 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:27.275748 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:27.277009 10149 hybrid_clock.cc:648] HybridClock initialized: now 1781894487276973 us; error 60 us; skew 500 ppm
I20260619 18:41:27.277390 10149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:27.280901 10149 init.cc:377] Logged in from keytab as oryx/127.3.85.188@KRBTEST.COM (short username oryx)
I20260619 18:41:27.282598 10149 webserver.cc:460] Webserver started at http://127.3.85.188:37241/ using document root <none> and password file <none>
I20260619 18:41:27.283330 10149 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:27.283438 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:27.283708 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:27.286973 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/instance:
uuid: "a2b0f75845594271afa674d5d93516f3"
format_stamp: "Formatted at 2026-06-19 18:41:27 on dist-test-slave-09fp"
server_key: "57f052e3e010ec0a61a0b92faedfafc5"
server_key_iv: "664511412d34041acf68965987c12f75"
server_key_version: "encryptionkey@0"
I20260619 18:41:27.287773 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal/instance:
uuid: "a2b0f75845594271afa674d5d93516f3"
format_stamp: "Formatted at 2026-06-19 18:41:27 on dist-test-slave-09fp"
server_key: "57f052e3e010ec0a61a0b92faedfafc5"
server_key_iv: "664511412d34041acf68965987c12f75"
server_key_version: "encryptionkey@0"
I20260619 18:41:27.293406 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:27.297158 10249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:27.298784 10149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:41:27.298939 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "a2b0f75845594271afa674d5d93516f3"
format_stamp: "Formatted at 2026-06-19 18:41:27 on dist-test-slave-09fp"
server_key: "57f052e3e010ec0a61a0b92faedfafc5"
server_key_iv: "664511412d34041acf68965987c12f75"
server_key_version: "encryptionkey@0"
I20260619 18:41:27.299119 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:27.314976 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:27.320657 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:27.321002 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:27.323391 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.315294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44598)
0619 18:41:27.315752 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.315792 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.315948 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.316522 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.316526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.316548 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:27.316697 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.316704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.318779 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.318785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.320261 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.320279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.321982 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.321988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.322247 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.322780 (+ 533us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.322946 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.323074 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":206,"threads_started":1}
I20260619 18:41:27.323557 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.315514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.3.85.189:45025)
0619 18:41:27.316116 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.316121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.316153 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:41:27.316213 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.316217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.316353 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.316483 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:27.316847 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.318573 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.320463 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.321778 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.321936 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.322249 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.322993 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.323138 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":294,"threads_started":1}
I20260619 18:41:27.325219 10195 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 18:41:27.325912 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6260ca280f49e888a6fdd9ad2024a9. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } }
I20260619 18:41:27.326268 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:41:27.343604 10149 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.188:33659
I20260619 18:41:27.343608 10303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.188:33659 every 8 connection(s)
I20260619 18:41:27.345103 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
I20260619 18:41:27.345530 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10149
I20260619 18:41:27.350569 10304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:27.360180 10304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
I20260619 18:41:27.380502 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.348326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.3.85.188:33659)
0619 18:41:27.349389 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.349413 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.351073 (+ 1660us) server_negotiation.cc:407] Connection header received
0619 18:41:27.353445 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.353476 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.354060 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.354575 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.356041 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.358156 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.359712 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.360471 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.360699 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.367768 (+ 7069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.367784 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.367818 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.367916 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.374049 (+ 6133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.375653 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.375670 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.375686 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.376060 (+ 374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.376700 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.376706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.376709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.377399 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.377801 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.378321 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.378950 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":447,"threads_started":1}
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.188@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:27.393929 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.363660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47702)
0619 18:41:27.376701 (+ 13041us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.376766 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.377008 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.377543 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.377551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.378423 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.378789 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.378801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.380495 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.380500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.382329 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.382345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.383590 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.383598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.383796 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.384953 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.385006 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.387316 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.391060 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.391074 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.391082 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.391672 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.392142 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.392145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.392148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.392349 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.392981 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.392988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.393147 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.393318 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.393628 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12715,"thread_start_us":12558,"threads_started":1}
I20260619 18:41:27.394032 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.363723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.3.85.190:45005)
0619 18:41:27.375998 (+ 12275us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.376005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.376873 (+ 868us) server_negotiation.cc:407] Connection header received
0619 18:41:27.377146 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.377152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.377226 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.377301 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.378990 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.380261 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.382549 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.383321 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.383489 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.387576 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.387582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.387586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.387625 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.390855 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.391864 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.391869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.391872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.392027 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.392522 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.392527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.392529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.392810 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.392989 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.393528 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.393732 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12109,"thread_start_us":12267,"threads_started":1}
I20260619 18:41:27.395771 10304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.188@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
I20260619 18:41:27.409878 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.396680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.3.85.189:45025)
0619 18:41:27.396883 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.396888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.396907 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:27.397001 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.397005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.397068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.397184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.397828 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.398678 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.399927 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.400385 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.400500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.404217 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.404223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.404227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.404265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.407043 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.407892 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.407896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.407899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.408049 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.408555 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.408560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.408562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.408871 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.408994 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.409351 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.409620 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:41:27.410324 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.396567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44610)
0619 18:41:27.396795 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.396811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.396902 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.397194 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.397197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.397537 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.397683 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.397690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.398824 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.398828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.399774 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.399786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.400546 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.400551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.400699 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.401466 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.401485 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.403928 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.407216 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.407221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.407224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.407706 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.408203 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.408207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.408209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.408369 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.408983 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.408987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.409116 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.409230 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.409542 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:41:27.411505 10304 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.188@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
I20260619 18:41:27.424697 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.412348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.0.0.1:57506)
0619 18:41:27.412538 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.412554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.412652 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.412914 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.412917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.413254 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.413421 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.413429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.414532 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.414536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.415397 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.415409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.416114 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.416120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.416317 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.416938 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.416955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.419249 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.422138 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.422146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.422150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.422619 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.423025 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.423032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.423035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.423187 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.423671 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.423675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.423820 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.423950 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.424237 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 18:41:27.424732 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.412259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.3.85.188:33659)
0619 18:41:27.412448 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.412453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.412619 (+ 166us) server_negotiation.cc:407] Connection header received
0619 18:41:27.412730 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.412733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.412788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.412877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.413560 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.414291 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.415526 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.415925 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.416067 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.419448 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.419454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.419459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.419497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.421975 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.422737 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.422744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.422747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.422913 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.423297 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.423302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.423305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.423556 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.423708 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.424041 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.424383 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":60}
I20260619 18:41:27.429574 10304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Bootstrap starting.
I20260619 18:41:27.432873 10304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:27.433986 10304 log.cc:824] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:27.437619 10304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: No bootstrap required, opened a new log
I20260619 18:41:27.445065 10304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } }
I20260619 18:41:27.445466 10304 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:27.445613 10304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b0f75845594271afa674d5d93516f3, State: Initialized, Role: FOLLOWER
I20260619 18:41:27.446244 10304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } }
I20260619 18:41:27.447156 10311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } } }
I20260619 18:41:27.447468 10311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:41:27.448271 10304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:41:27.458387 10322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:27.458603 10322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:27.463727 10149 master_runner.cc:428] Detected that this master a2b0f75845594271afa674d5d93516f3 is joining an existing cluster
I20260619 18:41:27.463845 10149 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.85.188:33659
I20260619 18:41:27.488493 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9646
I20260619 18:41:27.497004 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.468166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.3.85.189:45025)
0619 18:41:27.468390 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.468395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.475859 (+ 7464us) server_negotiation.cc:407] Connection header received
0619 18:41:27.476164 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.476171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.476277 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.476355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.477927 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.479094 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.481408 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.482085 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.482197 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.486277 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.486284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.486288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.486332 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.490066 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.492265 (+ 2199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.492276 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.492282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.492620 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.493389 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.493398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.493404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.493911 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.494102 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.496100 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.496467 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:41:27.498484 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.468922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:44616)
0619 18:41:27.475708 (+ 6786us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.475732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.475991 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.476594 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.476600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.477333 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.477732 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.477746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.479333 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.479339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.481174 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.481194 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.482573 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.482582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.482773 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.483562 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.483586 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.486049 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.490239 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.490246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.490256 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.490889 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.492847 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.492855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.492861 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.493163 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.494216 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.494222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.495644 (+ 1422us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.497307 (+ 1663us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.497790 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6617,"thread_start_us":12365,"threads_started":1}
I20260619 18:41:27.502408 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.467952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47716 (local address 127.3.85.190:45005)
0619 18:41:27.468150 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.468155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.484868 (+ 16713us) server_negotiation.cc:407] Connection header received
0619 18:41:27.485061 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.485067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.485157 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.485237 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.486831 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.488611 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.490717 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.491447 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.491554 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.494334 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.494341 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.494346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.494391 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.498571 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.499874 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.499881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.499885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.500052 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.500604 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.500612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.500617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.501001 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.501147 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.501775 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.502037 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 18:41:27.503774 10328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.474831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47716)
0619 18:41:27.484709 (+ 9878us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.484733 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.484882 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.485487 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.485491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.486198 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.486625 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.486638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.488805 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.488813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.490502 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.490521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.491852 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.491858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.492053 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.492821 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.492846 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.494016 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.499108 (+ 5092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.499114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.499118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.499685 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.500216 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.500222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.500228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.500466 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.501235 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.501240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.501468 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.502908 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.503255 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9225,"thread_start_us":836,"threads_started":1}
I20260619 18:41:27.515266 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.469575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.3.85.188:33659)
0619 18:41:27.469844 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.469849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.482046 (+ 12197us) server_negotiation.cc:407] Connection header received
0619 18:41:27.482258 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.482266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.482359 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.482509 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.484956 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.486309 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.501092 (+ 14783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.502110 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.502248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.506540 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.506553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.506557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.506594 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.511272 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.512362 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.512373 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.512380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.512547 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.513038 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.513045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.513051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.513528 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.513725 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.514374 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.514695 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.129@KRBTEST.COM
I20260619 18:41:27.524622 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.469102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.0.0.1:57512)
0619 18:41:27.481922 (+ 12820us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.481941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.482125 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.482731 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.482738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.483875 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.484725 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.484738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.487094 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.487099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.495397 (+ 8298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.495418 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.502641 (+ 7223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.502648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.502849 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.503824 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.505090 (+ 1266us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.506310 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.511462 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.511470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.511477 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.512201 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.512704 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.512712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.512718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.512916 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.513806 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.513814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.514023 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.523403 (+ 9380us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.523888 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12700,"thread_start_us":14801,"threads_started":1}
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894484, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.190@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
I20260619 18:41:27.530553 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.490407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.3.85.129:40031)
0619 18:41:27.504636 (+ 14229us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.504645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.504678 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:41:27.504759 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.504768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.505061 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.507171 (+ 2110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.507434 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.509527 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.512066 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.513041 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.519565 (+ 6524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.519887 (+ 322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.519902 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.519922 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.519957 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.524071 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.525097 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.525108 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.525111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.525301 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.526139 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.526144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.526148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.526647 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.529582 (+ 2935us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.529780 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.530076 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14039,"thread_start_us":13840,"threads_started":1}
I20260619 18:41:27.532132 9769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:27.537490 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.510060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.0.0.1:57520)
0619 18:41:27.510489 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.510510 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.510628 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.516089 (+ 5461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.516099 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.516717 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.517024 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.517041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.519930 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.519940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.521932 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.521956 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.525029 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.525040 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.525417 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.526129 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.526238 (+ 109us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.529481 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.534517 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.534527 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.534535 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.535257 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.535765 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.535770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.535774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.535926 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.536500 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.536505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.536659 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.536800 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.537106 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260619 18:41:27.537791 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.510849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.3.85.188:33659)
0619 18:41:27.515531 (+ 4682us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.515536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.515555 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:27.515673 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.515678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.515824 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.517297 (+ 1473us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.517579 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.519695 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.522214 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.523248 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.525833 (+ 2585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.529694 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.529702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.529711 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.529745 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.534117 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.535469 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.535477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.535482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.535630 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.536061 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.536066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.536069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.536367 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.536516 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.536898 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.537116 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4577,"thread_start_us":16189,"threads_started":1}
I20260619 18:41:27.539109 9585 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.85.188:33659: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a2b0f75845594271afa674d5d93516f3"
member_type: NON_VOTER
last_known_addr {
host: "127.3.85.188"
port: 33659
}
attrs {
promote: true
}
}
dest_uuid: "7e6260ca280f49e888a6fdd9ad2024a9"
cas_config_opid_index: 5
I20260619 18:41:27.540002 9585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } }
I20260619 18:41:27.542248 10254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2b0f75845594271afa674d5d93516f3 (127.3.85.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.546099 10195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:41:27.547962 10254 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.85.188:33659
I20260619 18:41:27.550648 10279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:27.550959 10149 master.cc:537] Master@127.3.85.188:33659 shutting down...
I20260619 18:41:27.562816 10335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260619 18:41:27.575660 10335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.576017 10335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:27.577872 10195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2b0f75845594271afa674d5d93516f3 (127.3.85.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.581276 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: NON_VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: true } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.581586 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: LEARNER
I20260619 18:41:27.582073 10254 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: attempting to promote NON_VOTER ddf9cd1c58a742ae918a390de219683e to VOTER
I20260619 18:41:27.582835 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } }
I20260619 18:41:27.585171 10195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 LEARNER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 18:41:27.586573 10334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260619 18:41:27.589318 10335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddf9cd1c58a742ae918a390de219683e (127.3.85.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.590303 10195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ddf9cd1c58a742ae918a390de219683e (127.3.85.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.593430 10335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.593729 10335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:27.593703 10254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.593926 10254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
W20260619 18:41:27.596385 10258 tablet.cc:2369] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:27.598593 10279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:41:27.599565 10223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.599936 10223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:41:27.605234 9570 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 -> Peer a2b0f75845594271afa674d5d93516f3 (127.3.85.188:33659): Couldn't send request to peer a2b0f75845594271afa674d5d93516f3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:41:27.606232 10149 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:27.606679 10149 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:27.606909 10149 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: stopping tablet replica
I20260619 18:41:27.623769 10149 master.cc:552] Master@127.3.85.188:33659 shutdown complete.
I20260619 18:41:27.623903 10149 master_runner.cc:305] Clearing existing system tablet
I20260619 18:41:27.624284 10149 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:27.624414 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:27.626969 10149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 18:41:27.628459 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:27.628811 10149 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:27.629089 10149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:41:27.629199 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "a2b0f75845594271afa674d5d93516f3"
format_stamp: "Formatted at 2026-06-19 18:41:27 on dist-test-slave-09fp"
server_key: "57f052e3e010ec0a61a0b92faedfafc5"
server_key_iv: "664511412d34041acf68965987c12f75"
server_key_version: "encryptionkey@0"
I20260619 18:41:27.632705 10149 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:27.637471 10149 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:41:27.637677 10149 log.cc:1196] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 18:41:27.638109 10149 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Deleting consensus metadata
I20260619 18:41:27.638365 10149 master_runner.cc:315] Copying system tablet from 127.3.85.190:45005
I20260619 18:41:27.640990 10149 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.85.190:45005
I20260619 18:41:27.655541 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.641828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.3.85.190:45005)
0619 18:41:27.642102 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:27.642111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:27.643085 (+ 974us) server_negotiation.cc:407] Connection header received
0619 18:41:27.643310 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:27.643316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:27.643411 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:27.643526 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:27.644391 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.645144 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.646432 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.646885 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.647024 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.649099 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:27.649104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:27.649108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:27.649140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:27.652136 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.653073 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.653080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.653084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.653296 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:27.653762 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:27.653770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:27.653773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:27.654124 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:27.654309 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:27.654843 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 18:41:27.655083 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 18:41:27.655650 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.641789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:47728)
0619 18:41:27.642958 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.642975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:27.643160 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:27.643551 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:27.643556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:27.644019 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:27.644228 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.644237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.645327 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.645333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.646279 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:27.646291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:27.647055 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:27.647062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:27.647243 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:27.647959 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:27.647978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:27.648868 (+ 890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:27.652310 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.652316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.652321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.652886 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.653444 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:27.653449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:27.653453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:27.653632 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:27.654265 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:27.654271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:27.654530 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 18:41:27.654724 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 18:41:27.655122 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":486,"threads_started":1}
I20260619 18:41:27.656704 9605 tablet_copy_service.cc:140] P 7e6260ca280f49e888a6fdd9ad2024a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a2b0f75845594271afa674d5d93516f3 ({username='oryx', principal='oryx/127.3.85.188@KRBTEST.COM'} at 127.0.0.1:47728)
I20260619 18:41:27.656898 9605 tablet_copy_service.cc:161] P 7e6260ca280f49e888a6fdd9ad2024a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a2b0f75845594271afa674d5d93516f3 at {username='oryx', principal='oryx/127.3.85.188@KRBTEST.COM'} at 127.0.0.1:47728: session id = a2b0f75845594271afa674d5d93516f3-00000000000000000000000000000000
I20260619 18:41:27.659857 9605 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:41:27.661826 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:41:27.668018 10149 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: tablet copy: Starting download of 0 data blocks...
I20260619 18:41:27.668396 10149 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: tablet copy: Starting download of 1 WAL segments...
I20260619 18:41:27.671047 10149 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:41:27.674710 9605 tablet_copy_service.cc:342] P 7e6260ca280f49e888a6fdd9ad2024a9: Request end of tablet copy session a2b0f75845594271afa674d5d93516f3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.85.188@KRBTEST.COM'} at 127.0.0.1:47728
I20260619 18:41:27.674885 9605 tablet_copy_service.cc:433] P 7e6260ca280f49e888a6fdd9ad2024a9: ending tablet copy session a2b0f75845594271afa674d5d93516f3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a2b0f75845594271afa674d5d93516f3
I20260619 18:41:27.679539 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9646
I20260619 18:41:27.690883 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:27.693972 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:27.699844 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:27.705462 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:27.706013 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:40031
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:45005
--webserver_port=41785
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:27.712049 10149 server_base.cc:1034] running on GCE node
I20260619 18:41:27.712460 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:27.713354 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:27.714533 10149 hybrid_clock.cc:648] HybridClock initialized: now 1781894487714479 us; error 74 us; skew 500 ppm
I20260619 18:41:27.714798 10149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:27 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894487, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:27.717409 10149 init.cc:377] Logged in from keytab as oryx/127.3.85.188@KRBTEST.COM (short username oryx)
I20260619 18:41:27.719707 10149 webserver.cc:460] Webserver started at http://127.3.85.188:34241/ using document root <none> and password file <none>
I20260619 18:41:27.720130 10149 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:27.720258 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:27.724071 10149 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:41:27.734972 10358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:27.735630 10149 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.009s
I20260619 18:41:27.735821 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "a2b0f75845594271afa674d5d93516f3"
format_stamp: "Formatted at 2026-06-19 18:41:27 on dist-test-slave-09fp"
server_key: "57f052e3e010ec0a61a0b92faedfafc5"
server_key_iv: "664511412d34041acf68965987c12f75"
server_key_version: "encryptionkey@0"
I20260619 18:41:27.736078 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:27.745436 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:27.750063 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:27.777058 10149 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.188:33659
I20260619 18:41:27.777081 10409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.188:33659 every 8 connection(s)
I20260619 18:41:27.778759 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
I20260619 18:41:27.784801 10410 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:41:27.789202 10410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Bootstrap starting.
I20260619 18:41:27.807811 10410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:41:27.808504 10410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: Bootstrap complete.
I20260619 18:41:27.809768 10410 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } }
I20260619 18:41:27.810160 10410 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b0f75845594271afa674d5d93516f3, State: Initialized, Role: LEARNER
I20260619 18:41:27.810510 10410 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } }
I20260619 18:41:27.811465 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:27.811774 10410 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:27.811918 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:41:27.830056 10424 catalog_manager.cc:1259] Loaded cluster ID: 6bf7eb60acdf4907a37fbb3e89abf98f
I20260619 18:41:27.830176 10424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: loading cluster ID for follower catalog manager: success
I20260619 18:41:27.834174 10424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: acquiring CA information for follower catalog manager: success
I20260619 18:41:27.837018 10424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:41:28.104316 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:28.104621 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:28.104719 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:28.112473 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:28.112605 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:28.113202 10353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:28.124382 10353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:40031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=41785
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:28.125826 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:28.127285 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:28.142071 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:28.142155 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:28.142132 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:28.142911 10353 server_base.cc:1034] running on GCE node
I20260619 18:41:28.143559 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:28.144601 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:28.145856 10353 hybrid_clock.cc:648] HybridClock initialized: now 1781894488145795 us; error 59 us; skew 500 ppm
I20260619 18:41:28.146210 10353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:28.149529 10353 init.cc:377] Logged in from keytab as oryx/127.3.85.129@KRBTEST.COM (short username oryx)
I20260619 18:41:28.151168 10353 webserver.cc:460] Webserver started at http://127.3.85.129:41785/ using document root <none> and password file <none>
I20260619 18:41:28.151952 10353 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:28.152127 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:28.157855 10353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:28.161209 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.163033 10353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:41:28.163252 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "fcaf9febf06f4bafa622d2bd2b53f49a"
format_stamp: "Formatted at 2026-06-19 18:41:24 on dist-test-slave-09fp"
server_key: "e1883a6a8930d91afc03b797a813b6e4"
server_key_iv: "5253aac5586b81b967bb7c2d447ff91f"
server_key_version: "encryptionkey@0"
I20260619 18:41:28.164773 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:28.175817 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.168973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.3.85.188:33659)
0619 18:41:28.169616 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.169623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.169648 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:28.169716 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.169719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.169785 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.169919 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:28.170341 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.171543 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.173849 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.174860 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.174986 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.175155 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.175449 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.175561 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":368,"threads_started":1}
I20260619 18:41:28.175856 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.168824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.0.0.1:57526)
0619 18:41:28.169309 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.169334 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.169487 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.169942 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.169946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.169976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:28.170175 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.170185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.171771 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.171777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.173610 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.173630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.175059 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.175065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.175263 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.175299 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.175403 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.175551 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":250,"threads_started":1}
I20260619 18:41:28.176983 10385 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 18:41:28.177670 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6260ca280f49e888a6fdd9ad2024a9. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:28.177974 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:41:28.197880 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:28.202811 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:28.203136 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:28.204169 10353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:28.205709 10353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:28.205857 10353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.205986 10353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:28.206079 10353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.243834 10353 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:40031
I20260619 18:41:28.243883 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:40031 every 8 connection(s)
I20260619 18:41:28.245425 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:28.247397 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10353
I20260619 18:41:28.247843 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9780
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.130@KRBTEST.COM
I20260619 18:41:28.299321 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.249855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:44507 (local address 127.3.85.188:33659)
0619 18:41:28.250111 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.250121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.273395 (+ 23274us) server_negotiation.cc:407] Connection header received
0619 18:41:28.276663 (+ 3268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.276689 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.276781 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.276892 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.280556 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.281605 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284641 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.285317 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.285432 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.290189 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.290197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.290205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.290253 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.294522 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.295576 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.295582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.295587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.295738 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.296422 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.296428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.296432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.297103 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.297346 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.298679 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.298945 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:41:28.300213 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.248971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.3.85.130:45429)
0619 18:41:28.277043 (+ 28072us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.277052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.277081 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:41:28.277146 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.277154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.277457 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.277627 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.279631 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.281681 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.283358 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.284261 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284396 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.291581 (+ 7185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.291603 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.291626 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.291677 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.295856 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.296853 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.296864 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.296867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.297045 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.297508 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.297513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.297516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.297965 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.298280 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.299527 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.299831 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27890,"thread_start_us":29542,"threads_started":1}
I20260619 18:41:28.300227 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.256218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:54603 (local address 127.3.85.190:45005)
0619 18:41:28.276523 (+ 20305us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.276530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.276548 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:28.276610 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.276616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.276689 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.277536 (+ 847us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.279258 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.280935 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.283485 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.284141 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284275 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.290805 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.290815 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.290821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.290869 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.295082 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.297619 (+ 2537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.297625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.297628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.297811 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.298286 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.298292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.298295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.298695 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.298840 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.299568 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.299833 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20167,"thread_start_us":20574,"threads_started":1}
I20260619 18:41:28.301011 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.249288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:39221 (local address 127.3.85.189:45025)
0619 18:41:28.274390 (+ 25102us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.274397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.274419 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:28.276409 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.276415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.276520 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.276656 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.279318 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.280398 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.283265 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.283962 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284102 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.289994 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.290003 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.290009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.290056 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.294066 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.295284 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.295303 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.295308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.295456 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.296515 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.296520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.296523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.297254 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.297441 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.299818 (+ 2377us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.300069 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24958,"thread_start_us":24827,"threads_started":1}
I20260619 18:41:28.301381 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.255560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:54603)
0619 18:41:28.273963 (+ 18403us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.273984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.276012 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.277159 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.277167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.278085 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.278994 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.279021 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.281121 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.281126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.283314 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.283326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.284553 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.284882 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.286151 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.286179 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.290531 (+ 4352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.295247 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.295253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.295256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.296384 (+ 1128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.297952 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.297959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.297962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.298133 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.298900 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.298904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.299046 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.299198 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.300746 (+ 1548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18242,"mutex_wait_us":156,"thread_start_us":14915,"threads_started":1}
I20260619 18:41:28.301728 9903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:28.301852 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.249590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.3.85.129:39221)
0619 18:41:28.269561 (+ 19971us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.269653 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.276012 (+ 6359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.277159 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.277167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.278901 (+ 1734us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.279167 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.279181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.280598 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.280603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.283059 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.283082 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.284465 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.284472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.284891 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.285856 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.285889 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.289634 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.294303 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.294315 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.294340 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.295132 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.296077 (+ 945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.296083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.296086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.296266 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.297505 (+ 1239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.297517 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.298184 (+ 667us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.298595 (+ 411us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.299201 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19446,"mutex_wait_us":196,"spinlock_wait_cycles":43648,"thread_start_us":23111,"threads_started":1}
I20260619 18:41:28.303311 10555 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:28.303635 10555 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.304422 10555 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:28.305482 10556 heartbeater.cc:347] Connected to a master server at 127.3.85.189:45025
I20260619 18:41:28.305619 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.250279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.3.85.129:44507)
0619 18:41:28.273228 (+ 22949us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.273255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.276500 (+ 3245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.277154 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.277167 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.279715 (+ 2548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.280335 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.280350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.281813 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.281823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.284445 (+ 2622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.284456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.285525 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.285533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.285771 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.286427 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.286480 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.289917 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.294724 (+ 4807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.294729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.294732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.295395 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.295926 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.295934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.295940 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.296255 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.297480 (+ 1225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.297519 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.298141 (+ 622us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.299589 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.299879 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22778,"mutex_wait_us":1109,"thread_start_us":23317,"threads_started":1}
I20260619 18:41:28.305677 10556 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.306210 10556 heartbeater.cc:511] Master 127.3.85.189:45025 requested a full tablet report, sending...
I20260619 18:41:28.306365 9585 ts_manager.cc:194] Re-registered known tserver with Master: fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:28.306805 10554 heartbeater.cc:347] Connected to a master server at 127.3.85.188:33659
I20260619 18:41:28.306936 10554 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.307385 10554 heartbeater.cc:511] Master 127.3.85.188:33659 requested a full tablet report, sending...
I20260619 18:41:28.307737 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:54603
I20260619 18:41:28.308341 10185 ts_manager.cc:194] Registered new tserver with Master: fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:28.309238 10375 ts_manager.cc:194] Registered new tserver with Master: fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:28.393886 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9780
I20260619 18:41:28.411044 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:45429
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:45005
--webserver_port=34341
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:28.709225 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: NON_VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: true } } }
I20260619 18:41:28.709765 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:28.713209 10567 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: attempting to promote NON_VOTER a2b0f75845594271afa674d5d93516f3 to VOTER
I20260619 18:41:28.714120 10567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } }
I20260619 18:41:28.716603 10385 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 LEARNER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:41:28.716935 10195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Refusing update from remote peer 7e6260ca280f49e888a6fdd9ad2024a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:41:28.717447 10334 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:41:28.717905 10569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:41:28.721438 10195 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b0f75845594271afa674d5d93516f3 (127.3.85.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.720759 10569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b0f75845594271afa674d5d93516f3 (127.3.85.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.729059 10569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.729501 10569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:28.729799 10334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.730023 10334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:28.731005 10573 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.731722 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.732060 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:28.732255 10385 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b0f75845594271afa674d5d93516f3 (127.3.85.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.731365 10573 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:41:28.738402 10575 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:28.738926 10575 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:41:28.774703 10566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:28.775012 10566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:28.775117 10566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:28.783183 10566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:28.783370 10566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:28.783557 10566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:28.795568 10566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:45429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=34341
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:28.796958 10566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:28.798246 10566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:28.812711 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:28.812781 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:28.812784 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:28.813787 10566 server_base.cc:1034] running on GCE node
I20260619 18:41:28.814718 10566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:28.815833 10566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:28.817090 10566 hybrid_clock.cc:648] HybridClock initialized: now 1781894488817063 us; error 46 us; skew 500 ppm
I20260619 18:41:28.817543 10566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:28.820994 10566 init.cc:377] Logged in from keytab as oryx/127.3.85.130@KRBTEST.COM (short username oryx)
I20260619 18:41:28.822796 10566 webserver.cc:460] Webserver started at http://127.3.85.130:34341/ using document root <none> and password file <none>
I20260619 18:41:28.823663 10566 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:28.823817 10566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:28.829864 10566 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:41:28.832931 10590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.834535 10566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:28.834765 10566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "b06c661541a74280b3a14c8c52242609"
format_stamp: "Formatted at 2026-06-19 18:41:25 on dist-test-slave-09fp"
server_key: "82ae468f88ce2594f0b43daab41cf75f"
server_key_iv: "adae3e4519ef08a92baf8f4b0376bc57"
server_key_version: "encryptionkey@0"
I20260619 18:41:28.835801 10566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:28.851634 10566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:28.855993 10566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:28.856309 10566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:28.857368 10566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:28.858829 10566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:28.858950 10566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.859086 10566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:28.859189 10566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:28.893172 10566 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:45429
I20260619 18:41:28.893210 10702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:45429 every 8 connection(s)
I20260619 18:41:28.894893 10566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:28.896081 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10566
I20260619 18:41:28.896361 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9914
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.131@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
Jun 19 18:41:28 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894488, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
I20260619 18:41:28.938439 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.897571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.3.85.131:46161)
0619 18:41:28.911214 (+ 13643us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.911229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.912017 (+ 788us) server_negotiation.cc:407] Connection header received
0619 18:41:28.912160 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.912172 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.912465 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.912655 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.915635 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.917780 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.919731 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.920648 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.921485 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.928563 (+ 7078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.928582 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.928602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.928661 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.933756 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.934713 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.934723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.934727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.934911 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.935469 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.935474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.935477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.935972 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.936320 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.937673 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.937976 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13457,"thread_start_us":13330,"threads_started":1}
I20260619 18:41:28.940547 10037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:28.944371 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.898025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:34385 (local address 127.3.85.190:45005)
0619 18:41:28.914961 (+ 16936us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.914971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.917847 (+ 2876us) server_negotiation.cc:407] Connection header received
0619 18:41:28.919948 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.919953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.920042 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.920138 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.925081 (+ 4943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.925899 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.928486 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.929202 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.929352 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.935713 (+ 6361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.935721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.935728 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.935786 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.939918 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.940967 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.940976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.940983 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.941209 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.941733 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.941739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.941743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.942099 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.942275 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.943688 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.943942 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16762,"thread_start_us":16955,"threads_started":1}
I20260619 18:41:28.945811 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.898100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51811 (local address 127.3.85.188:33659)
0619 18:41:28.910601 (+ 12501us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.910611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.918106 (+ 7495us) server_negotiation.cc:407] Connection header received
0619 18:41:28.919848 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.919856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.919942 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.920020 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.923509 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.924490 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.927625 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.928319 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.928435 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.935546 (+ 7111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.935556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.935563 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.935603 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.939097 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.940224 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.940230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.940234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.940408 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.941089 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.941094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.941097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.941915 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.942053 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.945125 (+ 3072us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.945411 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12297,"thread_start_us":12493,"threads_started":1}
I20260619 18:41:28.946485 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.898947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:34385)
0619 18:41:28.916743 (+ 17796us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.916799 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.919466 (+ 2667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.920572 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.920584 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.922145 (+ 1561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.923669 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.923681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.926041 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.926049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.928051 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.928070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.929656 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.929665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.929880 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.931011 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.931044 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.935457 (+ 4413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.940099 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.940105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.940109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.940761 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.941382 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.941390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.941396 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.941599 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.942253 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.942261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.943093 (+ 832us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.943536 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.944359 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17443,"mutex_wait_us":36,"spinlock_wait_cycles":768,"thread_start_us":19121,"threads_started":1}
I20260619 18:41:28.947396 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.899649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.3.85.130:51811)
0619 18:41:28.917966 (+ 18317us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.917993 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.919469 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.920566 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.920579 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.922854 (+ 2275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.923263 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.923286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.924671 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.924677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.927399 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.927417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.928776 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.928784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.929240 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.930151 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.930186 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.935260 (+ 5074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.939302 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.939313 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.939360 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.940046 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.940577 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.940581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.940583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.940949 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.942161 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.942171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.943152 (+ 981us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.946264 (+ 3112us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.946921 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17970,"mutex_wait_us":564,"spinlock_wait_cycles":536576,"thread_start_us":17787,"threads_started":1}
I20260619 18:41:28.948313 10704 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:28.946401 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.913249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.3.85.130:44907)
0619 18:41:28.918254 (+ 5005us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.918273 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:28.919466 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:28.921243 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:28.921248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:28.921885 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:28.923267 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.923291 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.924990 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.924996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.927399 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:28.927417 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.928776 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.928784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:28.929251 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.930151 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:28.930199 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:28.934062 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:28.942964 (+ 8902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.942969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.942973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.943585 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.944071 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:28.944076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:28.944080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:28.944252 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:28.945031 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:28.945036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:28.945201 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 18:41:28.945362 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 18:41:28.945644 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4872,"mutex_wait_us":165,"spinlock_wait_cycles":105088,"thread_start_us":3027,"threads_started":1}
I20260619 18:41:28.948740 10704 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.949570 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.913785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44907 (local address 127.3.85.189:45025)
0619 18:41:28.916052 (+ 2267us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:28.916061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:28.918398 (+ 2337us) server_negotiation.cc:407] Connection header received
0619 18:41:28.919938 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:28.919944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:28.920027 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:28.921006 (+ 979us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:28.923512 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.924826 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.927630 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:28.928341 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:28.928455 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:28.934389 (+ 5934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:28.934398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:28.934405 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:28.934488 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:28.942829 (+ 8341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.943753 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.943760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.943763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.943925 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:28.944400 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:28.944406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:28.944410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:28.944804 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:28.944967 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:28.948995 (+ 4028us) server_negotiation.cc:299] Negotiation successful
0619 18:41:28.949277 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2112,"thread_start_us":1987,"threads_started":1}
I20260619 18:41:28.949919 10704 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:28.950584 10703 heartbeater.cc:347] Connected to a master server at 127.3.85.188:33659
I20260619 18:41:28.950747 10703 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.951326 10703 heartbeater.cc:511] Master 127.3.85.188:33659 requested a full tablet report, sending...
I20260619 18:41:28.952184 10705 heartbeater.cc:347] Connected to a master server at 127.3.85.189:45025
I20260619 18:41:28.952361 10705 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:28.952927 10705 heartbeater.cc:511] Master 127.3.85.189:45025 requested a full tablet report, sending...
I20260619 18:41:28.953250 10375 ts_manager.cc:194] Registered new tserver with Master: b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429)
I20260619 18:41:28.954550 10185 ts_manager.cc:194] Registered new tserver with Master: b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429)
I20260619 18:41:28.955859 9585 ts_manager.cc:194] Re-registered known tserver with Master: b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429)
I20260619 18:41:28.957088 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:34385
I20260619 18:41:29.038528 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9914
I20260619 18:41:29.055022 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:46161
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:45005
--webserver_port=45427
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:29.310995 10555 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
W20260619 18:41:29.402949 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:29.403286 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:29.403383 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:29.411128 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:29.411269 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:29.411441 10716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:29.422796 10716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:46161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=45427
--tserver_master_addrs=127.3.85.190:45005,127.3.85.189:45025,127.3.85.188:33659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:29.424311 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:29.425726 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:29.439744 10722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:29.439841 10723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:29.439898 10725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:29.441825 10716 server_base.cc:1034] running on GCE node
I20260619 18:41:29.442546 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:29.443590 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:29.444825 10716 hybrid_clock.cc:648] HybridClock initialized: now 1781894489444755 us; error 73 us; skew 500 ppm
I20260619 18:41:29.445195 10716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:29 dist-test-slave-09fp krb5kdc[9540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894489, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:29.448945 10716 init.cc:377] Logged in from keytab as oryx/127.3.85.131@KRBTEST.COM (short username oryx)
I20260619 18:41:29.450748 10716 webserver.cc:460] Webserver started at http://127.3.85.131:45427/ using document root <none> and password file <none>
I20260619 18:41:29.451480 10716 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:29.451601 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:29.457825 10716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:41:29.461088 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:29.462963 10716 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:29.463212 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "88d46ba6f5064d47873f82f37d80a241"
format_stamp: "Formatted at 2026-06-19 18:41:26 on dist-test-slave-09fp"
server_key: "64250a675149da835d66bc6a453917f9"
server_key_iv: "761143b92b8e229a6a7495432246bd87"
server_key_version: "encryptionkey@0"
I20260619 18:41:29.464428 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:29.480794 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:29.485817 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:29.486156 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:29.487330 10716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:29.488824 10716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:29.488915 10716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:29.489080 10716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:29.489172 10716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:29.524675 10716 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46161
I20260619 18:41:29.524736 10844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46161 every 8 connection(s)
I20260619 18:41:29.526314 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:29.527738 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10716
Jun 19 18:41:29 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894489, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.188@KRBTEST.COM
Jun 19 18:41:29 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894489, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:29.580157 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.531346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.3.85.131:58619)
0619 18:41:29.538466 (+ 7120us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.538498 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.538692 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.547860 (+ 9168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.547881 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.549200 (+ 1319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:29.550402 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.550441 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.553174 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.553186 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.556405 (+ 3219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.556438 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.558110 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.558121 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.558593 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.559615 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:29.559660 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:29.563519 (+ 3859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:29.572281 (+ 8762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.572295 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.572348 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.573683 (+ 1335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.574494 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.574500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.574503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.574825 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.575611 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:29.575621 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:29.576164 (+ 543us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.576570 (+ 406us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.577222 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6934,"thread_start_us":6387,"threads_started":1}
I20260619 18:41:29.582041 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.531364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:58619 (local address 127.3.85.188:33659)
0619 18:41:29.546891 (+ 15527us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.546901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.546933 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:41:29.547025 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.547033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.547175 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.550748 (+ 3573us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.550992 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.552785 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.556783 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.557732 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.559208 (+ 1476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.563907 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.563919 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.563926 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.563985 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.571748 (+ 7763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.574008 (+ 2260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.574016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.574020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.574274 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.575025 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.575031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.575035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.575415 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.581078 (+ 5663us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.581258 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.581550 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15348,"thread_start_us":15245,"threads_started":1}
I20260619 18:41:29.584263 10845 heartbeater.cc:347] Connected to a master server at 127.3.85.188:33659
I20260619 18:41:29.584697 10845 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:29.585834 10845 heartbeater.cc:511] Master 127.3.85.188:33659 requested a full tablet report, sending...
I20260619 18:41:29.588256 10375 ts_manager.cc:194] Registered new tserver with Master: 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:29.589174 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.530341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:53209)
0619 18:41:29.533063 (+ 2722us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.533145 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.536507 (+ 3362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.565290 (+ 28783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.565300 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.565933 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:29.566315 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.566332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.572281 (+ 5949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.572289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.574505 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.574520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.575909 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.575918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.576139 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.577019 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:29.577047 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:29.580686 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:29.585754 (+ 5068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.585761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.585766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.586332 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.586851 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.586856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.586862 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.587072 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.588011 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:29.588017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:29.588153 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.588303 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.588660 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2290,"thread_start_us":1977,"threads_started":1}
I20260619 18:41:29.589928 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.529513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:53209 (local address 127.3.85.190:45005)
0619 18:41:29.562070 (+ 32557us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.562078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.562104 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:29.562187 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.562193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.562276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.566545 (+ 4269us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.566739 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.568185 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.574843 (+ 6658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.575609 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.578611 (+ 3002us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.581019 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.581027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.581032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.581091 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.585492 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.586522 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.586530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.586534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.586724 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.587304 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.587310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.587314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.587742 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.587947 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.588504 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.589646 (+ 1142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32380,"thread_start_us":33056,"threads_started":1}
I20260619 18:41:29.590948 10846 heartbeater.cc:347] Connected to a master server at 127.3.85.190:45005
I20260619 18:41:29.591137 10846 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:29.591564 10846 heartbeater.cc:511] Master 127.3.85.190:45005 requested a full tablet report, sending...
I20260619 18:41:29.592672 9585 ts_manager.cc:194] Re-registered known tserver with Master: 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:29.593601 9585 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:53209
Jun 19 18:41:29 dist-test-slave-09fp krb5kdc[9540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894489, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.189@KRBTEST.COM
I20260619 18:41:29.616268 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.539183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.3.85.131:36981)
0619 18:41:29.580732 (+ 41549us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.580755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.580871 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.596883 (+ 16012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.596887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.597295 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:29.597436 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.597446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.601585 (+ 4139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.601593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.603073 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.603089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.604065 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.604071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.604721 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.605470 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:29.605494 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:29.608727 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:29.613054 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.613063 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.613066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.613703 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.614277 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:29.614283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:29.614287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:29.614495 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:29.615259 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:29.615267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:29.615435 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.615573 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.615889 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41394,"thread_start_us":60545,"threads_started":1}
I20260619 18:41:29.618775 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.539240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:36981 (local address 127.3.85.189:45025)
0619 18:41:29.596325 (+ 57085us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.596332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.596354 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:29.596418 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.596423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.596540 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.596664 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.597632 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.601386 (+ 3754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.603249 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.603920 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.604043 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.609090 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.609098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.609103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.609152 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.612856 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.613919 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.613925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.613930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.614107 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.614627 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.614632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.614637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.615099 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.616574 (+ 1475us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.616723 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.617073 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56961,"thread_start_us":60894,"threads_started":1}
I20260619 18:41:29.619727 10847 heartbeater.cc:347] Connected to a master server at 127.3.85.189:45025
I20260619 18:41:29.619910 10847 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:29.620467 10847 heartbeater.cc:511] Master 127.3.85.189:45025 requested a full tablet report, sending...
I20260619 18:41:29.621830 10185 ts_manager.cc:194] Registered new tserver with Master: 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:29.623570 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.562609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.3.85.189:45025)
0619 18:41:29.600699 (+ 38090us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.600705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.600724 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:29.600777 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.600782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.600860 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.602385 (+ 1525us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.605322 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.606192 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.608201 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.608611 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.608798 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.613123 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.613132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.613137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.613246 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.618320 (+ 5074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.620122 (+ 1802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.620127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.620131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.620282 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.620744 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.620749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.620752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.621177 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.621450 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.621913 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.622136 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37956,"thread_start_us":33475,"threads_started":1}
I20260619 18:41:29.623621 10862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.562794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.3.85.190:45005)
0619 18:41:29.570257 (+ 7463us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.570267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.595229 (+ 24962us) server_negotiation.cc:407] Connection header received
0619 18:41:29.595566 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.595572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.595658 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.595818 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.603970 (+ 8152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.604884 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.610230 (+ 5346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.611040 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.611216 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.615932 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.615943 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.615951 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.616005 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.619548 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.620749 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.620755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.620758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.620909 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.621377 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.621382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.621384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.621731 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.622018 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.622902 (+ 884us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.623188 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7225,"thread_start_us":6920,"threads_started":1}
I20260619 18:41:29.625931 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.598721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.3.85.188:33659)
0619 18:41:29.600394 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.600400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.603728 (+ 3328us) server_negotiation.cc:407] Connection header received
0619 18:41:29.603905 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.603912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.604012 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.604134 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:29.607438 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.608527 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.612108 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.612815 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.612960 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.618119 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:29.618127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:29.618133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:29.618181 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:29.622961 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.623618 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.623623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.623819 (+ 196us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.623976 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:29.624402 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:29.624407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:29.624411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:29.624785 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:29.625036 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.625389 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.625614 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536}
I20260619 18:41:29.628430 9585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:29.636363 9585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:29.673621 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.657420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.3.85.129:40031)
0619 18:41:29.667002 (+ 9582us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.667012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.667052 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:41:29.667155 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.667161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.667373 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.667631 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:29.668289 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.669512 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.671541 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.672510 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.672699 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.672937 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.673191 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.673351 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9414,"thread_start_us":9287,"threads_started":1}
I20260619 18:41:29.673628 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.656866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40031 (local address 127.0.0.1:38878)
0619 18:41:29.663764 (+ 6898us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.663783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.663909 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.667623 (+ 3714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.667629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.667660 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:29.668082 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.668096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.669728 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.669732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.671368 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.671384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.672771 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.672779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.672955 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.673000 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.673121 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.673267 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6758,"thread_start_us":6553,"threads_started":1}
I20260619 18:41:29.679190 10489 tablet_service.cc:1467] Processing CreateTablet for tablet c1c4895daf0146cf813f96f329b4fadf (DEFAULT_TABLE table=test-table [id=52288b17c8674856bbe008ba13b9a5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:29.680884 10489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c4895daf0146cf813f96f329b4fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:29.683400 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.657274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45429 (local address 127.0.0.1:34496)
0619 18:41:29.674077 (+ 16803us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.674096 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.674220 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.677246 (+ 3026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.677253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.677286 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:29.677482 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.677490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.679471 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.679476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.681345 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.681396 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.682794 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.682800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.682953 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.682981 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.683042 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.683157 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16683,"thread_start_us":21432,"threads_started":1}
I20260619 18:41:29.684461 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.657490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.3.85.130:45429)
0619 18:41:29.676398 (+ 18908us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.676407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.676436 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:41:29.676539 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.676546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.676774 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.677229 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:29.677632 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.679300 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.681574 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.682570 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.683718 (+ 1148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.683923 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.683992 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.684174 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18748,"thread_start_us":19439,"threads_started":1}
I20260619 18:41:29.687007 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.660386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.3.85.131:46161)
0619 18:41:29.676050 (+ 15664us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.676057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.676093 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:41:29.676293 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.676300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.676514 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.676763 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:29.677736 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.679317 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.683976 (+ 4659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.685828 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.686013 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.686252 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.686525 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.686696 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15514,"thread_start_us":18121,"threads_started":1}
I20260619 18:41:29.686993 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.660013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46161 (local address 127.0.0.1:40544)
0619 18:41:29.675387 (+ 15374us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.675408 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.675537 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.676748 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.676752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.676784 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:29.677141 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.677152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.681596 (+ 4444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.681601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.683638 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.683662 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.686070 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.686076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.686298 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.686336 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.686449 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.686624 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15249,"thread_start_us":15245,"threads_started":1}
I20260619 18:41:29.688429 10638 tablet_service.cc:1467] Processing CreateTablet for tablet c1c4895daf0146cf813f96f329b4fadf (DEFAULT_TABLE table=test-table [id=52288b17c8674856bbe008ba13b9a5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:29.689810 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c4895daf0146cf813f96f329b4fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:29.692891 10780 tablet_service.cc:1467] Processing CreateTablet for tablet c1c4895daf0146cf813f96f329b4fadf (DEFAULT_TABLE table=test-table [id=52288b17c8674856bbe008ba13b9a5f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:29.694499 10780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c4895daf0146cf813f96f329b4fadf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:29.696974 10878 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Bootstrap starting.
I20260619 18:41:29.701192 10878 tablet_bootstrap.cc:654] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:29.702356 10878 log.cc:824] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:29.705251 10878 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: No bootstrap required, opened a new log
I20260619 18:41:29.705552 10878 ts_tablet_manager.cc:1397] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 18:41:29.707353 10880 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Bootstrap starting.
I20260619 18:41:29.710507 10881 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Bootstrap starting.
I20260619 18:41:29.711448 10880 tablet_bootstrap.cc:654] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:29.712774 10880 log.cc:824] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:29.714377 10881 tablet_bootstrap.cc:654] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:29.716447 10881 log.cc:824] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:29.716822 10880 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: No bootstrap required, opened a new log
I20260619 18:41:29.717128 10880 ts_tablet_manager.cc:1397] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 18:41:29.716933 10878 raft_consensus.cc:348] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.717433 10878 raft_consensus.cc:374] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:29.717557 10878 raft_consensus.cc:729] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaf9febf06f4bafa622d2bd2b53f49a, State: Initialized, Role: FOLLOWER
I20260619 18:41:29.718230 10878 consensus_queue.cc:260] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.720335 10878 ts_tablet_manager.cc:1428] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 18:41:29.722402 10881 tablet_bootstrap.cc:492] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: No bootstrap required, opened a new log
I20260619 18:41:29.722754 10881 ts_tablet_manager.cc:1397] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260619 18:41:29.727594 10880 raft_consensus.cc:348] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.728086 10880 raft_consensus.cc:374] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:29.728193 10880 raft_consensus.cc:729] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06c661541a74280b3a14c8c52242609, State: Initialized, Role: FOLLOWER
I20260619 18:41:29.729074 10880 consensus_queue.cc:260] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.732654 10704 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
I20260619 18:41:29.733261 10880 ts_tablet_manager.cc:1428] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260619 18:41:29.733605 10881 raft_consensus.cc:348] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.734112 10881 raft_consensus.cc:374] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:29.734217 10881 raft_consensus.cc:729] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d46ba6f5064d47873f82f37d80a241, State: Initialized, Role: FOLLOWER
I20260619 18:41:29.735008 10881 consensus_queue.cc:260] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.742146 10846 heartbeater.cc:503] Master 127.3.85.190:45005 was elected leader, sending a full tablet report...
I20260619 18:41:29.743496 10881 ts_tablet_manager.cc:1428] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Time spent starting tablet: real 0.021s user 0.017s sys 0.002s
W20260619 18:41:29.776367 10557 tablet.cc:2369] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:29.793474 10853 tablet.cc:2369] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:29.913501 10706 tablet.cc:2369] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:29.960213 10884 raft_consensus.cc:482] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:29.961005 10884 raft_consensus.cc:504] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.963179 10884 leader_election.cc:288] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429), 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:29.972052 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.963775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:36401 (local address 127.3.85.130:45429)
0619 18:41:29.964039 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.964049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.964875 (+ 826us) server_negotiation.cc:407] Connection header received
0619 18:41:29.965011 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.965017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.965090 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.965188 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:29.965934 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.966793 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.969510 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.970933 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.971119 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.971219 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.971619 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.971759 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:41:29.972289 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.963646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:40703 (local address 127.3.85.131:46161)
0619 18:41:29.963850 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:29.963857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:29.964598 (+ 741us) server_negotiation.cc:407] Connection header received
0619 18:41:29.964900 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:29.964906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:29.964996 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:29.965119 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:29.965758 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.966839 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.969878 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.971149 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.971314 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.971364 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:29.971898 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 18:41:29.972041 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:41:29.972575 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.963425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45429 (local address 127.3.85.129:36401)
0619 18:41:29.964716 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.964731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.964895 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.965343 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.965347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.965367 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:29.965756 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.965766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.966951 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.966959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.969273 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.969313 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.971184 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.971192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.971399 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.971447 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.971570 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.971727 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":784,"thread_start_us":380,"threads_started":1}
I20260619 18:41:29.973016 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.963425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46161 (local address 127.3.85.129:40703)
0619 18:41:29.964469 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.964489 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:29.964723 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:29.965136 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:29.965141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:29.965176 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:29.965576 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.965588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.967004 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.967010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.969688 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:29.969708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:29.971498 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:29.971503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:29.971705 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:29.971740 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:41:29.972458 (+ 718us) client_negotiation.cc:240] Negotiation successful
0619 18:41:29.972590 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":43,"thread_start_us":618,"threads_started":1}
I20260619 18:41:29.974072 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c4895daf0146cf813f96f329b4fadf" candidate_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06c661541a74280b3a14c8c52242609" is_pre_election: true
I20260619 18:41:29.974664 10658 raft_consensus.cc:2454] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcaf9febf06f4bafa622d2bd2b53f49a in term 0.
I20260619 18:41:29.975482 10444 leader_election.cc:302] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06c661541a74280b3a14c8c52242609, fcaf9febf06f4bafa622d2bd2b53f49a; no voters:
I20260619 18:41:29.975936 10884 raft_consensus.cc:2781] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:29.976090 10884 raft_consensus.cc:482] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:29.975951 10800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c4895daf0146cf813f96f329b4fadf" candidate_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d46ba6f5064d47873f82f37d80a241" is_pre_election: true
I20260619 18:41:29.976229 10884 raft_consensus.cc:3037] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:29.976470 10800 raft_consensus.cc:2454] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcaf9febf06f4bafa622d2bd2b53f49a in term 0.
I20260619 18:41:29.978979 10884 raft_consensus.cc:504] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.979727 10884 leader_election.cc:288] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [CANDIDATE]: Term 1 election: Requested vote from peers b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429), 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161)
I20260619 18:41:29.980374 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c4895daf0146cf813f96f329b4fadf" candidate_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06c661541a74280b3a14c8c52242609"
I20260619 18:41:29.980350 10800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c4895daf0146cf813f96f329b4fadf" candidate_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d46ba6f5064d47873f82f37d80a241"
I20260619 18:41:29.980592 10658 raft_consensus.cc:3037] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:29.980615 10800 raft_consensus.cc:3037] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:29.982962 10658 raft_consensus.cc:2454] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcaf9febf06f4bafa622d2bd2b53f49a in term 1.
I20260619 18:41:29.983183 10800 raft_consensus.cc:2454] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcaf9febf06f4bafa622d2bd2b53f49a in term 1.
I20260619 18:41:29.983479 10444 leader_election.cc:302] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b06c661541a74280b3a14c8c52242609, fcaf9febf06f4bafa622d2bd2b53f49a; no voters:
I20260619 18:41:29.983771 10884 raft_consensus.cc:2781] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:29.984218 10884 raft_consensus.cc:686] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 LEADER]: Becoming Leader. State: Replica: fcaf9febf06f4bafa622d2bd2b53f49a, State: Running, Role: LEADER
I20260619 18:41:29.984814 10884 consensus_queue.cc:237] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:29.990808 9584 catalog_manager.cc:5495] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a reported cstate change: term changed from 0 to 1, leader changed from <none> to fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129). New cstate: current_term: 1 leader_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:30.063393 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:30.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.3.85.129:40031)
0619 18:41:30.058589 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:30.058593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:30.058616 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:41:30.058701 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:30.058705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:30.058791 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:30.058921 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:30.059246 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.059939 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.061239 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.061642 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.061736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:30.062313 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:30.062877 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:30.062993 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:30.063105 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 18:41:30.063181 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:41:30.072681 10658 raft_consensus.cc:1261] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Refusing update from remote peer fcaf9febf06f4bafa622d2bd2b53f49a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:30.072679 10800 raft_consensus.cc:1261] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Refusing update from remote peer fcaf9febf06f4bafa622d2bd2b53f49a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:30.073724 10889 consensus_queue.cc:1035] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:30.074129 10884 consensus_queue.cc:1035] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:30.089474 10892 mvcc.cc:205] Tried to move back new op lower bound from 7298639831326597120 to 7298639830981898240. Current Snapshot: MvccSnapshot[applied={T|T < 7298639831326597120}]
I20260619 18:41:30.107949 9584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:30.108260 9584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:30.120018 9584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6260ca280f49e888a6fdd9ad2024a9: Sending DeleteTablet for 3 replicas of tablet c1c4895daf0146cf813f96f329b4fadf
I20260619 18:41:30.121254 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c4895daf0146cf813f96f329b4fadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:30 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:34496
I20260619 18:41:30.124087 10489 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c4895daf0146cf813f96f329b4fadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:30 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260619 18:41:30.125751 10780 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c4895daf0146cf813f96f329b4fadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:30 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40544
I20260619 18:41:30.125805 10779 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c4895daf0146cf813f96f329b4fadf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:30 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40544
I20260619 18:41:30.126848 10905 tablet_replica.cc:330] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: stopping tablet replica
I20260619 18:41:30.127414 10905 raft_consensus.cc:2229] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:30.128063 10905 raft_consensus.cc:2258] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:30.130475 10905 ts_tablet_manager.cc:1905] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:30.135218 10902 tablet_replica.cc:330] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: stopping tablet replica
I20260619 18:41:30.135835 10902 raft_consensus.cc:2229] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:30.136658 10902 raft_consensus.cc:2258] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:30.139365 10902 ts_tablet_manager.cc:1905] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:30.139714 10905 ts_tablet_manager.cc:1918] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:30.140445 10905 log.cc:1196] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/c1c4895daf0146cf813f96f329b4fadf
I20260619 18:41:30.141254 10905 ts_tablet_manager.cc:1939] T c1c4895daf0146cf813f96f329b4fadf P 88d46ba6f5064d47873f82f37d80a241: Deleting consensus metadata
I20260619 18:41:30.143489 9570 catalog_manager.cc:4841] TS 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161): tablet c1c4895daf0146cf813f96f329b4fadf (table test-table [id=52288b17c8674856bbe008ba13b9a5f6]) successfully deleted
W20260619 18:41:30.144162 9570 catalog_manager.cc:4808] TS 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161): delete failed for tablet c1c4895daf0146cf813f96f329b4fadf because the tablet was not found. No further retry: Not found: Tablet not found: c1c4895daf0146cf813f96f329b4fadf
I20260619 18:41:30.149158 10902 ts_tablet_manager.cc:1918] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:30.149401 10902 log.cc:1196] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/c1c4895daf0146cf813f96f329b4fadf
I20260619 18:41:30.150161 10902 ts_tablet_manager.cc:1939] T c1c4895daf0146cf813f96f329b4fadf P b06c661541a74280b3a14c8c52242609: Deleting consensus metadata
I20260619 18:41:30.150888 10904 tablet_replica.cc:330] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: stopping tablet replica
I20260619 18:41:30.151607 10904 raft_consensus.cc:2229] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:30.152395 10904 raft_consensus.cc:2258] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:30.154982 9571 catalog_manager.cc:4841] TS b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429): tablet c1c4895daf0146cf813f96f329b4fadf (table test-table [id=52288b17c8674856bbe008ba13b9a5f6]) successfully deleted
I20260619 18:41:30.157636 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 9553
I20260619 18:41:30.159055 9615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:30.161854 10904 ts_tablet_manager.cc:1905] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:30.171244 10904 ts_tablet_manager.cc:1918] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:30.171481 10904 log.cc:1196] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/c1c4895daf0146cf813f96f329b4fadf
I20260619 18:41:30.172102 10904 ts_tablet_manager.cc:1939] T c1c4895daf0146cf813f96f329b4fadf P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting consensus metadata
I20260619 18:41:30.175866 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:30.123536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.3.85.189:45025)
0619 18:41:30.123721 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:30.123727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:30.150094 (+ 26367us) server_negotiation.cc:407] Connection header received
0619 18:41:30.152364 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:30.152374 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:30.152511 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:30.152684 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:30.155495 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.156620 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.162327 (+ 5707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.163064 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.163188 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:30.167644 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:30.167653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:30.167659 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:30.167721 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:30.171893 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.172891 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.172897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.172901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.173095 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.173628 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.173635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.173638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.174059 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:30.174320 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:30.175204 (+ 884us) server_negotiation.cc:299] Negotiation successful
0619 18:41:30.175486 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:41:30.195024 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:30.150346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.3.85.188:33659)
0619 18:41:30.162072 (+ 11726us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:30.162081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:30.176121 (+ 14040us) server_negotiation.cc:407] Connection header received
0619 18:41:30.176199 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:30.176205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:30.176285 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:30.176398 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:30.179212 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.180408 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.182587 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.183304 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.183462 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:30.187286 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:30.187294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:30.187300 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:30.187353 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:30.191241 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.192251 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.192258 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.192262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.192435 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.192892 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.192897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.192901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.193331 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:30.193590 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:30.194344 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 18:41:30.194616 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11584,"thread_start_us":11732,"threads_started":1}
I20260619 18:41:30.298612 9569 catalog_manager.cc:4841] TS fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031): tablet c1c4895daf0146cf813f96f329b4fadf (table test-table [id=52288b17c8674856bbe008ba13b9a5f6]) successfully deleted
I20260619 18:41:30.299393 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 9553
I20260619 18:41:30.358171 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:30.326858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.3.85.189:45025)
0619 18:41:30.327055 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:30.327063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:30.341245 (+ 14182us) server_negotiation.cc:407] Connection header received
0619 18:41:30.341551 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:30.341557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:30.341643 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:30.341722 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:30.344384 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.345385 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.346939 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.347590 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.347744 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:30.351424 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:30.351431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:30.351435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:30.351477 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:30.354982 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.355995 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.355999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.356002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.356150 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.356615 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.356620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.356624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.357035 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:30.357306 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:30.357645 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:41:30.357856 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:41:30.359421 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:30.326862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.3.85.188:33659)
0619 18:41:30.333571 (+ 6709us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:30.333578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:30.340700 (+ 7122us) server_negotiation.cc:407] Connection header received
0619 18:41:30.341396 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:30.341406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:30.341510 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:30.341596 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:30.343958 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.345028 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.347124 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:30.347778 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:30.347896 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:30.351008 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:30.351013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:30.351017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:30.351057 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:30.354982 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.356724 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.356732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.356738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.356880 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:30.357332 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:30.357338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:30.357341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:30.357720 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:30.358153 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:30.358811 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 18:41:30.359077 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6530}
I20260619 18:41:31.101910 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.101012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:43191)
0619 18:41:31.101560 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.101706 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
W20260619 18:41:31.102564 10704 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:45005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.105983 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.105129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:56455)
0619 18:41:31.105631 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.105743 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
W20260619 18:41:31.106501 10555 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:45005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.123152 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.122213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:33963)
0619 18:41:31.122787 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.122921 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":317,"threads_started":1}
W20260619 18:41:31.123548 10846 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:45005 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.619431 10918 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e6260ca280f49e888a6fdd9ad2024a9)
I20260619 18:41:31.619738 10918 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } }
I20260619 18:41:31.621207 10918 leader_election.cc:288] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005), a2b0f75845594271afa674d5d93516f3 (127.3.85.188:33659)
I20260619 18:41:31.621379 10919 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7e6260ca280f49e888a6fdd9ad2024a9)
I20260619 18:41:31.621709 10919 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } }
I20260619 18:41:31.623440 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.621492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42010)
0619 18:41:31.622959 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.623132 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1186,"threads_started":1}
W20260619 18:41:31.624047 10171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.624577 10919 leader_election.cc:288] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005), ddf9cd1c58a742ae918a390de219683e (127.3.85.189:45025)
I20260619 18:41:31.627990 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.623860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42018)
0619 18:41:31.627448 (+ 3588us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.627678 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3442,"thread_start_us":3287,"threads_started":1}
W20260619 18:41:31.628294 10361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.629063 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.628482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42026)
0619 18:41:31.628738 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.628864 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260619 18:41:31.629393 10171 leader_election.cc:334] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005): Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.633298 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.630696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42028)
0619 18:41:31.631168 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.631362 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
W20260619 18:41:31.633517 10361 leader_election.cc:334] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005): Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.635581 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.621703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.3.85.188:33659)
0619 18:41:31.622784 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.622792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.622813 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:41:31.622933 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.622941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.623038 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.623154 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.623842 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.631317 (+ 7475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.633127 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.634598 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.634735 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.634889 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.635180 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.635318 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":935,"thread_start_us":780,"threads_started":1}
I20260619 18:41:31.635648 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.621401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:33659 (local address 127.0.0.1:42886)
0619 18:41:31.621967 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.622000 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.622174 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.623206 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.623212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.623248 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.623653 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.623670 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.631469 (+ 7799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.631476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.632797 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.632813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.634792 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.634799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.635011 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.635052 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.635161 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.635310 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":419,"threads_started":1}
I20260619 18:41:31.636363 10385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf9cd1c58a742ae918a390de219683e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2b0f75845594271afa674d5d93516f3" is_pre_election: true
I20260619 18:41:31.636773 10385 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf9cd1c58a742ae918a390de219683e in term 1.
I20260619 18:41:31.637362 10170 leader_election.cc:302] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2b0f75845594271afa674d5d93516f3, ddf9cd1c58a742ae918a390de219683e; no voters: 7e6260ca280f49e888a6fdd9ad2024a9
I20260619 18:41:31.637840 10918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:41:31.638013 10918 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7e6260ca280f49e888a6fdd9ad2024a9)
I20260619 18:41:31.638125 10918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:41:31.639439 10926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.625065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.3.85.189:45025)
0619 18:41:31.630994 (+ 5929us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.630999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.631018 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:31.631077 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.631080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.631134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.631254 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.633478 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.634608 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.637097 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.638524 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.638628 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.638719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.639046 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.639151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5772,"thread_start_us":5599,"threads_started":1}
I20260619 18:41:31.639508 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.624780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45025 (local address 127.0.0.1:55214)
0619 18:41:31.630227 (+ 5447us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.630249 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.630416 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.631287 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.631295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.631330 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.633292 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.633304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.634759 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.634765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.636897 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.636930 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.638696 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.638703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.638879 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.638920 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.639018 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.639165 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5336,"mutex_wait_us":20,"thread_start_us":6630,"threads_started":1}
I20260619 18:41:31.640218 10195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2b0f75845594271afa674d5d93516f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ddf9cd1c58a742ae918a390de219683e" is_pre_election: true
I20260619 18:41:31.641147 10918 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } }
I20260619 18:41:31.641628 10195 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2b0f75845594271afa674d5d93516f3 in current term 2: Already voted for candidate ddf9cd1c58a742ae918a390de219683e in this term.
I20260619 18:41:31.642763 10359 leader_election.cc:302] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2b0f75845594271afa674d5d93516f3; no voters: 7e6260ca280f49e888a6fdd9ad2024a9, ddf9cd1c58a742ae918a390de219683e
I20260619 18:41:31.643121 10918 leader_election.cc:288] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 election: Requested vote from peers 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005), a2b0f75845594271afa674d5d93516f3 (127.3.85.188:33659)
I20260619 18:41:31.643141 10919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:41:31.643709 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.643276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42044)
0619 18:41:31.643472 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.643570 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:41:31.643757 10385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddf9cd1c58a742ae918a390de219683e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2b0f75845594271afa674d5d93516f3"
I20260619 18:41:31.644703 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.644310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42050)
0619 18:41:31.644459 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.644554 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:41:31.644863 10171 leader_election.cc:334] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005): Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
I20260619 18:41:31.645745 10919 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:31.648027 10385 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf9cd1c58a742ae918a390de219683e in term 2.
I20260619 18:41:31.649125 10170 leader_election.cc:302] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2b0f75845594271afa674d5d93516f3, ddf9cd1c58a742ae918a390de219683e; no voters: 7e6260ca280f49e888a6fdd9ad2024a9
I20260619 18:41:31.649493 10918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:41:31.649951 10918 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [term 2 LEADER]: Becoming Leader. State: Replica: ddf9cd1c58a742ae918a390de219683e, State: Running, Role: LEADER
I20260619 18:41:31.650331 10918 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } }
I20260619 18:41:31.652159 10928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf9cd1c58a742ae918a390de219683e. Latest consensus state: current_term: 2 leader_uuid: "ddf9cd1c58a742ae918a390de219683e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:31.652652 10928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:31.653550 10930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:31.657387 10930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:31.658295 10930 catalog_manager.cc:1259] Loaded cluster ID: 6bf7eb60acdf4907a37fbb3e89abf98f
I20260619 18:41:31.658396 10930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:31.659603 10930 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:31.660454 10930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Loaded TSK: 0
I20260619 18:41:31.661398 10930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:31.705963 10185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:31.713279 10185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:31.717995 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.717551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42082)
0619 18:41:31.717710 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.717799 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:41:31.718027 10385 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [term 2 FOLLOWER]: Refusing update from remote peer ddf9cd1c58a742ae918a390de219683e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 18:41:31.718879 10928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 18:41:31.719266 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.718791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42096)
0619 18:41:31.719001 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.719095 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:41:31.719446 10171 consensus_peers.cc:487] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e -> Peer 7e6260ca280f49e888a6fdd9ad2024a9 (127.3.85.190:45005): Couldn't send request to peer 7e6260ca280f49e888a6fdd9ad2024a9. Status: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:41:31.722035 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf9cd1c58a742ae918a390de219683e. Latest consensus state: current_term: 2 leader_uuid: "ddf9cd1c58a742ae918a390de219683e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:31.722438 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:41:31.724036 10928 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ddf9cd1c58a742ae918a390de219683e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:31.724360 10928 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:31.725157 10919 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ddf9cd1c58a742ae918a390de219683e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7e6260ca280f49e888a6fdd9ad2024a9" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 45005 } } peers { permanent_uuid: "ddf9cd1c58a742ae918a390de219683e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45025 } attrs { promote: false } } peers { permanent_uuid: "a2b0f75845594271afa674d5d93516f3" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 33659 } attrs { promote: false } } }
I20260619 18:41:31.725379 10932 mvcc.cc:205] Tried to move back new op lower bound from 7298639838069796864 to 7298639837803974656. Current Snapshot: MvccSnapshot[applied={T|T < 7298639838069796864}]
I20260619 18:41:31.725457 10919 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2b0f75845594271afa674d5d93516f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:41:31.747872 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.734865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40031 (local address 127.0.0.1:34804)
0619 18:41:31.735285 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.735312 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.735491 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.737674 (+ 2183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.737683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.737725 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.738218 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.738237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.740682 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.740693 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.744049 (+ 3356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.744077 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.747056 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.747066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.747311 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.747347 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.747422 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.747589 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 18:41:31.748767 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.734859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.3.85.129:40031)
0619 18:41:31.737165 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.737176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.737208 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:41:31.737313 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.737322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.737434 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.738429 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.738702 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.740394 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.745017 (+ 4623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.746701 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.748311 (+ 1610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.748392 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.748440 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.748550 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"thread_start_us":3104,"threads_started":1}
I20260619 18:41:31.749517 10489 tablet_service.cc:1467] Processing CreateTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 (DEFAULT_TABLE table=test-table [id=50df2e373e4e4b6e8bf8630131a4403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:31.750034 10489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14ce82ce75d4c3e973b8dbbbc170e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:31.755604 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.736676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46161 (local address 127.0.0.1:52472)
0619 18:41:31.741758 (+ 5082us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.741782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.741930 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.749813 (+ 7883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.749821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.749861 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.750251 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.750266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.751880 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.751885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.753422 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.753436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.754738 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.754744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.754940 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.754990 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.755062 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.755227 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4932,"thread_start_us":4252,"threads_started":1}
I20260619 18:41:31.756600 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.733945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45429 (local address 127.0.0.1:33520)
0619 18:41:31.734208 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.734246 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.734391 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.750532 (+ 16141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.750538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.750564 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.750752 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.750762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.752249 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.752254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.754187 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.754205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.755919 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.755927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.756110 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.756145 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.756222 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.756356 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 18:41:31.756639 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.737147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.3.85.131:46161)
0619 18:41:31.749255 (+ 12108us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.749265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.749292 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:41:31.749385 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.749393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.749501 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.750421 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.750617 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.751718 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.753614 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.754491 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.756033 (+ 1542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.756145 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.756202 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.756347 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11909,"thread_start_us":12455,"threads_started":1}
I20260619 18:41:31.757043 10941 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Bootstrap starting.
I20260619 18:41:31.757332 10780 tablet_service.cc:1467] Processing CreateTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 (DEFAULT_TABLE table=test-table [id=50df2e373e4e4b6e8bf8630131a4403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:31.757772 10780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14ce82ce75d4c3e973b8dbbbc170e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:31.758661 10941 tablet_bootstrap.cc:654] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:31.763469 10943 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Bootstrap starting.
I20260619 18:41:31.763558 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.734001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.3.85.130:45429)
0619 18:41:31.750058 (+ 16057us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.750064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.750082 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:31.750154 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.750159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.750240 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.750886 (+ 646us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.751035 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.752097 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.754398 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.755697 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.762987 (+ 7290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.763123 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.763182 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.763302 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15923,"thread_start_us":16325,"threads_started":1}
I20260619 18:41:31.764349 10638 tablet_service.cc:1467] Processing CreateTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 (DEFAULT_TABLE table=test-table [id=50df2e373e4e4b6e8bf8630131a4403d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:31.764827 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d14ce82ce75d4c3e973b8dbbbc170e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:31.766314 10943 tablet_bootstrap.cc:654] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:31.766355 10941 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: No bootstrap required, opened a new log
I20260619 18:41:31.766522 10941 ts_tablet_manager.cc:1397] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260619 18:41:31.767442 10941 raft_consensus.cc:348] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.767707 10941 raft_consensus.cc:374] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:31.767797 10941 raft_consensus.cc:729] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaf9febf06f4bafa622d2bd2b53f49a, State: Initialized, Role: FOLLOWER
I20260619 18:41:31.768088 10941 consensus_queue.cc:260] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.768710 10943 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: No bootstrap required, opened a new log
I20260619 18:41:31.768901 10943 ts_tablet_manager.cc:1397] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:41:31.769620 10941 ts_tablet_manager.cc:1428] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:31.774124 10943 raft_consensus.cc:348] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.774376 10943 raft_consensus.cc:374] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:31.774492 10943 raft_consensus.cc:729] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d46ba6f5064d47873f82f37d80a241, State: Initialized, Role: FOLLOWER
I20260619 18:41:31.774755 10943 consensus_queue.cc:260] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.775872 10556 heartbeater.cc:503] Master 127.3.85.189:45025 was elected leader, sending a full tablet report...
I20260619 18:41:31.776379 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.769435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:44921)
0619 18:41:31.775855 (+ 6420us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.776041 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6240,"thread_start_us":5914,"threads_started":1}
I20260619 18:41:31.780519 10947 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Bootstrap starting.
I20260619 18:41:31.782181 10943 ts_tablet_manager.cc:1428] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20260619 18:41:31.782418 10947 tablet_bootstrap.cc:654] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:31.782573 10847 heartbeater.cc:503] Master 127.3.85.189:45025 was elected leader, sending a full tablet report...
I20260619 18:41:31.784885 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.781419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:49095)
0619 18:41:31.784457 (+ 3038us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.784590 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2892,"thread_start_us":2736,"threads_started":1}
I20260619 18:41:31.789265 10947 tablet_bootstrap.cc:492] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: No bootstrap required, opened a new log
I20260619 18:41:31.789424 10947 ts_tablet_manager.cc:1397] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20260619 18:41:31.790181 10947 raft_consensus.cc:348] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.790500 10947 raft_consensus.cc:374] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:31.790601 10947 raft_consensus.cc:729] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b06c661541a74280b3a14c8c52242609, State: Initialized, Role: FOLLOWER
I20260619 18:41:31.790853 10947 consensus_queue.cc:260] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.791706 10947 ts_tablet_manager.cc:1428] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 18:41:31.802167 10705 heartbeater.cc:503] Master 127.3.85.189:45025 was elected leader, sending a full tablet report...
I20260619 18:41:31.804225 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.777372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:51903)
0619 18:41:31.803696 (+ 26324us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.803910 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26175,"thread_start_us":26015,"threads_started":1}
I20260619 18:41:31.804877 10948 raft_consensus.cc:482] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:31.805078 10948 raft_consensus.cc:504] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.805495 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.792798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:37485)
0619 18:41:31.805030 (+ 12232us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.805213 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12080,"thread_start_us":11963,"threads_started":1}
I20260619 18:41:31.806663 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.806088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:37625)
0619 18:41:31.806260 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.806342 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 18:41:31.807150 10948 leader_election.cc:288] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429), fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:31.807420 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.806907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:57099)
0619 18:41:31.807148 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.807249 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 18:41:31.814826 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.808022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:37431 (local address 127.3.85.129:40031)
0619 18:41:31.808257 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.808266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.808293 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:41:31.808390 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.808398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.808517 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.808660 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.809486 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.810430 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.812650 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.813702 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.813897 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.813979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.814316 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.814501 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:41:31.814870 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.807346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:40031 (local address 127.3.85.131:37431)
0619 18:41:31.807604 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.807625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.807792 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.808733 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.808737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.808784 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.809276 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.809290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.810635 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.810642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.812457 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.812472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.813897 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.813903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.814117 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.814151 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.814262 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.814414 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"spinlock_wait_cycles":10624,"thread_start_us":741,"threads_started":1}
I20260619 18:41:31.815694 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.807406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:41979 (local address 127.3.85.130:45429)
0619 18:41:31.807774 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.807792 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.808945 (+ 1153us) server_negotiation.cc:407] Connection header received
0619 18:41:31.809226 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.809234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.809311 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.809407 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:31.810101 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.811153 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.813335 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.814414 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.814609 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.814687 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.815278 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.815432 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260619 18:41:31.815757 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.807358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45429 (local address 127.3.85.131:41979)
0619 18:41:31.808037 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.808827 (+ 790us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.809012 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.809555 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.809562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.809589 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:31.809901 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.809912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.811339 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.811376 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.813147 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.813163 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.814844 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.814857 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.815079 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.815127 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.815230 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.815387 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"spinlock_wait_cycles":1408,"thread_start_us":1956,"threads_started":1}
I20260619 18:41:31.815802 10509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14ce82ce75d4c3e973b8dbbbc170e55" candidate_uuid: "88d46ba6f5064d47873f82f37d80a241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" is_pre_election: true
I20260619 18:41:31.816254 10509 raft_consensus.cc:2454] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88d46ba6f5064d47873f82f37d80a241 in term 0.
I20260619 18:41:31.816695 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14ce82ce75d4c3e973b8dbbbc170e55" candidate_uuid: "88d46ba6f5064d47873f82f37d80a241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06c661541a74280b3a14c8c52242609" is_pre_election: true
I20260619 18:41:31.817025 10658 raft_consensus.cc:2454] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88d46ba6f5064d47873f82f37d80a241 in term 0.
I20260619 18:41:31.817339 10733 leader_election.cc:302] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88d46ba6f5064d47873f82f37d80a241, fcaf9febf06f4bafa622d2bd2b53f49a; no voters:
I20260619 18:41:31.817906 10948 raft_consensus.cc:2781] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:31.818022 10948 raft_consensus.cc:482] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:31.818086 10948 raft_consensus.cc:3037] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:31.820381 10948 raft_consensus.cc:504] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.820983 10948 leader_election.cc:288] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [CANDIDATE]: Term 1 election: Requested vote from peers b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429), fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031)
I20260619 18:41:31.821630 10658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14ce82ce75d4c3e973b8dbbbc170e55" candidate_uuid: "88d46ba6f5064d47873f82f37d80a241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b06c661541a74280b3a14c8c52242609"
I20260619 18:41:31.821867 10658 raft_consensus.cc:3037] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:31.822072 10509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d14ce82ce75d4c3e973b8dbbbc170e55" candidate_uuid: "88d46ba6f5064d47873f82f37d80a241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a"
I20260619 18:41:31.822322 10509 raft_consensus.cc:3037] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:31.824913 10658 raft_consensus.cc:2454] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88d46ba6f5064d47873f82f37d80a241 in term 1.
I20260619 18:41:31.825587 10735 leader_election.cc:302] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88d46ba6f5064d47873f82f37d80a241, b06c661541a74280b3a14c8c52242609; no voters:
I20260619 18:41:31.825593 10509 raft_consensus.cc:2454] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88d46ba6f5064d47873f82f37d80a241 in term 1.
I20260619 18:41:31.826058 10948 raft_consensus.cc:2781] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:31.826705 10948 raft_consensus.cc:686] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 1 LEADER]: Becoming Leader. State: Replica: 88d46ba6f5064d47873f82f37d80a241, State: Running, Role: LEADER
I20260619 18:41:31.827302 10948 consensus_queue.cc:237] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } }
I20260619 18:41:31.827768 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.827299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:60247)
0619 18:41:31.827485 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.827617 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:41:31.831322 10185 catalog_manager.cc:5495] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131). New cstate: current_term: 1 leader_uuid: "88d46ba6f5064d47873f82f37d80a241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88d46ba6f5064d47873f82f37d80a241" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46161 } health_report { overall_health: HEALTHY } } }
I20260619 18:41:31.863461 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.858229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.3.85.131:46161)
0619 18:41:31.858510 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.858517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.858540 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:41:31.858653 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.858657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.858732 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.858903 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:31.859259 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.859970 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.861260 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.861675 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.861809 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.862231 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:31.862870 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:31.863041 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.863092 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.863177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:41:31.868042 10658 raft_consensus.cc:1261] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Refusing update from remote peer 88d46ba6f5064d47873f82f37d80a241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:31.868404 10509 raft_consensus.cc:1261] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Refusing update from remote peer 88d46ba6f5064d47873f82f37d80a241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:31.869067 10956 consensus_queue.cc:1035] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b06c661541a74280b3a14c8c52242609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:31.869514 10948 consensus_queue.cc:1035] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcaf9febf06f4bafa622d2bd2b53f49a" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:31.872550 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.871901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:55925)
0619 18:41:31.872190 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.872313 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 18:41:31.872568 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.872071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:45691)
0619 18:41:31.872318 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.872416 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 18:41:31.880097 10953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.879452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.130:57267)
0619 18:41:31.879801 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.879897 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260619 18:41:31.885383 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.883615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.129:35395)
0619 18:41:31.884910 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.885092 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1104}
I20260619 18:41:31.887193 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.883258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.3.85.131:57407)
0619 18:41:31.886666 (+ 3408us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.886804 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:41:31.897735 10184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:31.898036 10184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:31.904402 10184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddf9cd1c58a742ae918a390de219683e: Sending DeleteTablet for 3 replicas of tablet d14ce82ce75d4c3e973b8dbbbc170e55
I20260619 18:41:31.905663 10638 tablet_service.cc:1514] Processing DeleteTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:31 UTC) from {username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:33520
I20260619 18:41:31.905901 10489 tablet_service.cc:1514] Processing DeleteTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:31 UTC) from {username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:34804
I20260619 18:41:31.909577 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10353
I20260619 18:41:31.910179 10969 tablet_replica.cc:330] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: stopping tablet replica
I20260619 18:41:31.910699 10969 raft_consensus.cc:2229] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:31.911129 10969 raft_consensus.cc:2258] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:31.913619 10969 ts_tablet_manager.cc:1905] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:31.919335 10780 tablet_service.cc:1514] Processing DeleteTablet for tablet d14ce82ce75d4c3e973b8dbbbc170e55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:31 UTC) from {username='oryx', principal='oryx/127.3.85.189@KRBTEST.COM'} at 127.0.0.1:52472
I20260619 18:41:31.929497 10971 tablet_replica.cc:330] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: stopping tablet replica
I20260619 18:41:31.929826 10971 raft_consensus.cc:2229] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:31.930825 10971 raft_consensus.cc:2258] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:31.931537 10968 tablet_replica.cc:330] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: stopping tablet replica
I20260619 18:41:31.931768 10968 raft_consensus.cc:2229] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:31.932075 10968 raft_consensus.cc:2258] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:31.933065 10971 ts_tablet_manager.cc:1905] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:31.935531 10968 ts_tablet_manager.cc:1905] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:31.936446 10969 ts_tablet_manager.cc:1918] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:31.939429 10969 log.cc:1196] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/d14ce82ce75d4c3e973b8dbbbc170e55
I20260619 18:41:31.939955 10969 ts_tablet_manager.cc:1939] T d14ce82ce75d4c3e973b8dbbbc170e55 P fcaf9febf06f4bafa622d2bd2b53f49a: Deleting consensus metadata
I20260619 18:41:31.940425 10971 ts_tablet_manager.cc:1918] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:31.940577 10971 log.cc:1196] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/d14ce82ce75d4c3e973b8dbbbc170e55
I20260619 18:41:31.941182 10971 ts_tablet_manager.cc:1939] T d14ce82ce75d4c3e973b8dbbbc170e55 P 88d46ba6f5064d47873f82f37d80a241: Deleting consensus metadata
I20260619 18:41:31.942481 10170 catalog_manager.cc:4841] TS 88d46ba6f5064d47873f82f37d80a241 (127.3.85.131:46161): tablet d14ce82ce75d4c3e973b8dbbbc170e55 (table test-table [id=50df2e373e4e4b6e8bf8630131a4403d]) successfully deleted
I20260619 18:41:31.942781 10169 catalog_manager.cc:4841] TS fcaf9febf06f4bafa622d2bd2b53f49a (127.3.85.129:40031): tablet d14ce82ce75d4c3e973b8dbbbc170e55 (table test-table [id=50df2e373e4e4b6e8bf8630131a4403d]) successfully deleted
I20260619 18:41:31.943368 10968 ts_tablet_manager.cc:1918] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:31.943478 10968 log.cc:1196] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/d14ce82ce75d4c3e973b8dbbbc170e55
I20260619 18:41:31.943809 10968 ts_tablet_manager.cc:1939] T d14ce82ce75d4c3e973b8dbbbc170e55 P b06c661541a74280b3a14c8c52242609: Deleting consensus metadata
I20260619 18:41:31.944869 10171 catalog_manager.cc:4841] TS b06c661541a74280b3a14c8c52242609 (127.3.85.130:45429): tablet d14ce82ce75d4c3e973b8dbbbc170e55 (table test-table [id=50df2e373e4e4b6e8bf8630131a4403d]) successfully deleted
I20260619 18:41:31.951063 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.911748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.3.85.129:40031)
0619 18:41:31.912030 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.912040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.918556 (+ 6516us) server_negotiation.cc:407] Connection header received
0619 18:41:31.919728 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.919735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.919834 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.919918 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.929842 (+ 9924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.931027 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.933149 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.933812 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.933967 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.943221 (+ 9254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.943241 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.943264 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.943333 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.947706 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.948580 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.948592 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.948596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.948760 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.949188 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.949193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.949197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.949699 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.949986 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.950334 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.950650 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 18:41:31.951788 10549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:32.072459 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10353
I20260619 18:41:32.104599 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10566
I20260619 18:41:32.121624 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.105473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.3.85.130:45429)
0619 18:41:32.105645 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.105650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.105668 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:32.105863 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.105867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.105929 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.106013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:32.107835 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.108639 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.110601 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.111028 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.111146 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.115070 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:32.115088 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:32.115105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:32.115157 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:32.118423 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:32.119508 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:32.119516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:32.119519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:32.119671 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:32.120089 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:32.120092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:32.120095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:32.120495 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:32.120721 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.121061 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.121247 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:41:32.122385 10698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:32.132066 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.131534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42110)
0619 18:41:32.131753 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.131865 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:41:32.133448 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.133010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:45005 (local address 127.0.0.1:42116)
0619 18:41:32.133187 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.133279 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:45005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:41:32.219106 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10566
I20260619 18:41:32.237587 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10716
I20260619 18:41:32.254422 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.238379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.3.85.131:46161)
0619 18:41:32.238569 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.238578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.238603 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:32.238813 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.238818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.238904 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.239023 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:32.240559 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.241368 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.242841 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.243288 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.243454 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.247233 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:32.247249 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:32.247266 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:32.247315 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:32.251173 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:32.252049 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:32.252057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:32.252059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:32.252238 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:32.252687 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:32.252692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:32.252694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:32.253100 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:32.253469 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.253763 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.254016 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:41:32.255306 10840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:32.382522 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10716
I20260619 18:41:32.404311 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10048
I20260619 18:41:32.405512 10215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:32.527652 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10048
I20260619 18:41:32.549332 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10149
I20260619 18:41:32.550573 10405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:32.664594 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10149
2026-06-19T18:41:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10096 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:32 dist-test-slave-09fp krb5kdc[10979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:32 dist-test-slave-09fp krb5kdc[10979](info): set up 2 sockets
Jun 19 18:41:32 dist-test-slave-09fp krb5kdc[10979](info): commencing operation
krb5kdc: starting...
W20260619 18:41:33.779599 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:33 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab.
Entry for principal oryx/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab.
I20260619 18:41:34.015847 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37513
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab
--principal=oryx/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:34.359684 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:34.360075 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:34.360239 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:34.367872 10992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:34.368050 10992 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:41:34.368156 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:34.368286 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:34.368376 10992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:34.368467 10992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:34.378746 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37513
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.190.keytab
--principal=oryx/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:34.380324 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:34.381875 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:34.395545 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:34.395614 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:34.395594 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:34.396520 10992 server_base.cc:1034] running on GCE node
I20260619 18:41:34.397468 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:34.399192 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:34.400514 10992 hybrid_clock.cc:648] HybridClock initialized: now 1781894494400452 us; error 60 us; skew 500 ppm
I20260619 18:41:34.400938 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:34 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894494, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:34.404616 10992 init.cc:377] Logged in from keytab as oryx/127.3.85.190@KRBTEST.COM (short username oryx)
I20260619 18:41:34.406639 10992 webserver.cc:460] Webserver started at http://127.3.85.190:38755/ using document root <none> and password file <none>
I20260619 18:41:34.407370 10992 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:34.407469 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:34.407843 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:34.411083 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "6d2b56687eca4caab52a417d189d71c5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-09fp"
server_key: "fcbc48afd5180742f89228c4a4f983e1"
server_key_iv: "392a771d0b9627c199807a5b802818cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.411964 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "6d2b56687eca4caab52a417d189d71c5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-09fp"
server_key: "fcbc48afd5180742f89228c4a4f983e1"
server_key_iv: "392a771d0b9627c199807a5b802818cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.417569 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:34.421484 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:34.423154 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:34.423377 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "6d2b56687eca4caab52a417d189d71c5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-09fp"
server_key: "fcbc48afd5180742f89228c4a4f983e1"
server_key_iv: "392a771d0b9627c199807a5b802818cd"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.423635 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:34.439348 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:34.443801 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:34.444097 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:34.467195 10992 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37513
I20260619 18:41:34.467172 11068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37513 every 8 connection(s)
I20260619 18:41:34.469357 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:34.473654 11069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:34.479189 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 10992
I20260619 18:41:34.479403 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:34.480196 3414 external_mini_cluster.cc:1428] Setting key d6966285ff322d68d2b802ee8ed3a9cb
I20260619 18:41:34.485824 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Bootstrap starting.
I20260619 18:41:34.489265 11069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:34.490373 11069 log.cc:824] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:34.493723 11069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: No bootstrap required, opened a new log
Jun 19 18:41:34 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:34.502002 11069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } }
I20260619 18:41:34.502475 11069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:34.502620 11069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2b56687eca4caab52a417d189d71c5, State: Initialized, Role: FOLLOWER
I20260619 18:41:34.503340 11069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } }
I20260619 18:41:34.503623 11069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:34.503788 11069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:34.503981 11069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:34.504283 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:34.482581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.3.85.190:37513)
0619 18:41:34.483486 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:34.483498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:34.483543 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:34.485282 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:34.485306 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:34.485737 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:34.486182 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:34.487760 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:34.489113 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:34.490557 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:34.491431 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:34.491676 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:34.495916 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:34.495924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:34.495943 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:34.495992 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:34.499518 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:34.500404 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:34.500412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:34.500420 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:34.500621 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:34.501042 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:34.501046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:34.501048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:34.501643 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:34.501960 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:34.502471 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 18:41:34.502941 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":431,"threads_started":1}
I20260619 18:41:34.507015 11069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } }
I20260619 18:41:34.507668 11069 leader_election.cc:302] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d2b56687eca4caab52a417d189d71c5; no voters:
I20260619 18:41:34.508198 11069 leader_election.cc:288] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:34.508345 11074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:34.509115 11074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [term 1 LEADER]: Becoming Leader. State: Replica: 6d2b56687eca4caab52a417d189d71c5, State: Running, Role: LEADER
I20260619 18:41:34.509712 11074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } }
I20260619 18:41:34.510550 11069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:34.513595 11075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d2b56687eca4caab52a417d189d71c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } } }
I20260619 18:41:34.514389 11075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:34.514717 11076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d2b56687eca4caab52a417d189d71c5. Latest consensus state: current_term: 1 leader_uuid: "6d2b56687eca4caab52a417d189d71c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d2b56687eca4caab52a417d189d71c5" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37513 } } }
I20260619 18:41:34.515033 11076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:34.517789 11082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:34.522257 11082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:34.531945 11082 catalog_manager.cc:1347] Generated new cluster ID: 1e8040099899415d85c1dffcac25c691
I20260619 18:41:34.532156 11082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:34.571262 11082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:34.572672 11082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:34.584705 11082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Generated new TSK 0
I20260619 18:41:34.585656 11082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
I20260619 18:41:34.647259 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37513
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:35.020475 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:35.020798 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:35.020900 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:35.028586 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:35.028723 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:35.028810 11095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:41:35.028923 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:35.040095 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:35.041410 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:35.042758 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:35.056488 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.056550 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.056571 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.058197 11095 server_base.cc:1034] running on GCE node
I20260619 18:41:35.059005 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:35.060175 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:35.063740 11095 hybrid_clock.cc:648] HybridClock initialized: now 1781894495063606 us; error 76 us; skew 500 ppm
I20260619 18:41:35.064177 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:35 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:35.067893 11095 init.cc:377] Logged in from keytab as oryx/127.3.85.129@KRBTEST.COM (short username oryx)
I20260619 18:41:35.069818 11095 webserver.cc:460] Webserver started at http://127.3.85.129:34977/ using document root <none> and password file <none>
I20260619 18:41:35.070710 11095 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:35.070820 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:35.071105 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:35.074154 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "36ef2e84b2204305a21f1b292b0fd930"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "e9bcb4baac6b7932f7f28e1941e9bd1d"
server_key_iv: "1772ce8c5adef0dbe3c7aa87ca83cf5b"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.075088 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "36ef2e84b2204305a21f1b292b0fd930"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "e9bcb4baac6b7932f7f28e1941e9bd1d"
server_key_iv: "1772ce8c5adef0dbe3c7aa87ca83cf5b"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.081240 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:41:35.085376 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.087044 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:41:35.087266 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "36ef2e84b2204305a21f1b292b0fd930"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "e9bcb4baac6b7932f7f28e1941e9bd1d"
server_key_iv: "1772ce8c5adef0dbe3c7aa87ca83cf5b"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.087630 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:35.113387 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:35.118562 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:35.118877 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:35.121956 11095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:35.122130 11095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.122308 11095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:35.122394 11095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:41:35 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:35.156066 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.123027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:51865 (local address 127.3.85.190:37513)
0619 18:41:35.123557 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.123564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.125184 (+ 1620us) server_negotiation.cc:407] Connection header received
0619 18:41:35.128147 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.128154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.128262 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.128420 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.131338 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.132520 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.136336 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.136826 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.136973 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.143857 (+ 6884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.143868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.143877 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.143941 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.147689 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.149501 (+ 1812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.149508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.149511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.149984 (+ 473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.150818 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.150828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.150835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.151325 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.151494 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.155057 (+ 3563us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.155489 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
I20260619 18:41:35.156526 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.123860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.129:51865)
0619 18:41:35.124937 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.124993 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.127781 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.128734 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.128748 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.129797 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.131044 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.131099 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.132843 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.132851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.136043 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.136092 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.137207 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.137222 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.137666 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.138847 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.138893 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.143408 (+ 4515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.148067 (+ 4659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.148085 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.148111 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.149082 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.150211 (+ 1129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.150218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.150222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.150623 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.151481 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.151490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.152204 (+ 714us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.152659 (+ 455us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.153829 (+ 1170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":475,"threads_started":1}
I20260619 18:41:35.171765 11095 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:45125
I20260619 18:41:35.171980 11230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:45125 every 8 connection(s)
I20260619 18:41:35.173843 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:35.175529 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11095
I20260619 18:41:35.175774 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:35.176666 11231 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37513
I20260619 18:41:35.176700 3414 external_mini_cluster.cc:1428] Setting key c3969e9086415318ddd8a4336bc39737
I20260619 18:41:35.176949 11231 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:35.177904 11231 heartbeater.cc:511] Master 127.3.85.190:37513 requested a full tablet report, sending...
I20260619 18:41:35.180833 11024 ts_manager.cc:194] Registered new tserver with Master: 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125)
I20260619 18:41:35.182737 11024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:51865
I20260619 18:41:35.186712 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.165102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.129:47551)
0619 18:41:35.167320 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.167344 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.167560 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.168318 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.168327 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.169174 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.169638 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.169657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.171803 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.171813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.173541 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.173563 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.174683 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.174695 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.174991 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.176386 (+ 1395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.176418 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.177992 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.183090 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.183097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.183101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.183727 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.184311 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.184319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.184325 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.184504 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.185265 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.185272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.185509 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.185680 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.186074 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"thread_start_us":1905,"threads_started":1}
I20260619 18:41:35.186905 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.164862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:47551 (local address 127.3.85.190:37513)
0619 18:41:35.165140 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.165151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.167508 (+ 2357us) server_negotiation.cc:407] Connection header received
0619 18:41:35.167764 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.167775 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.167894 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.168035 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.170088 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.171560 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.173781 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.174328 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.174580 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.178294 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.178300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.178304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.178340 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.182898 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.183948 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.183954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.183958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.184141 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.184650 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.184659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.184665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.185085 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.185250 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.185901 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.186314 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
WARNING: no policy specified for oryx/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
I20260619 18:41:35.230746 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37513
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:35.583492 11235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:35.583809 11235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:35.583909 11235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:35.591781 11235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:35.591993 11235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:35.592145 11235 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:41:35.592288 11235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:35.602908 11235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:35.604271 11235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:35.605650 11235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:35.620150 11240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.620182 11243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.620226 11241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.621212 11235 server_base.cc:1034] running on GCE node
I20260619 18:41:35.621901 11235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:35.623049 11235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:35.624294 11235 hybrid_clock.cc:648] HybridClock initialized: now 1781894495624198 us; error 71 us; skew 500 ppm
I20260619 18:41:35.624637 11235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:35 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:35.628162 11235 init.cc:377] Logged in from keytab as oryx/127.3.85.130@KRBTEST.COM (short username oryx)
I20260619 18:41:35.629869 11235 webserver.cc:460] Webserver started at http://127.3.85.130:33675/ using document root <none> and password file <none>
I20260619 18:41:35.630735 11235 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:35.630878 11235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:35.631229 11235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:35.634428 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "3a6f73c544e848be83a8527106fcfed9"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "4af3e7ba72566819a7cacf0f9ee4cfa5"
server_key_iv: "74a938aad82d1fd2565fbb7343a422f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.635411 11235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "3a6f73c544e848be83a8527106fcfed9"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "4af3e7ba72566819a7cacf0f9ee4cfa5"
server_key_iv: "74a938aad82d1fd2565fbb7343a422f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.640982 11235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:35.644816 11250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.646281 11235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:35.646548 11235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "3a6f73c544e848be83a8527106fcfed9"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-09fp"
server_key: "4af3e7ba72566819a7cacf0f9ee4cfa5"
server_key_iv: "74a938aad82d1fd2565fbb7343a422f8"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.646818 11235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:35.680836 11235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:35.685448 11235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:35.685779 11235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:35.688777 11235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:35.688937 11235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.689123 11235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:35.689219 11235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:41:35 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:35.716439 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.689771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45523 (local address 127.3.85.190:37513)
0619 18:41:35.690218 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.690224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.691737 (+ 1513us) server_negotiation.cc:407] Connection header received
0619 18:41:35.695185 (+ 3448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.695194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.695311 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.695438 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.698007 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.699306 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.701903 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.702826 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.703017 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.708251 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.708256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.708260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.708296 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.712100 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.713507 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.713515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.713520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.713670 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.714396 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.714401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.714403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.714761 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.714922 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.715762 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.716010 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260619 18:41:35.717557 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.690548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.130:45523)
0619 18:41:35.691522 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.691573 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.694682 (+ 3109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.695844 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.695862 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.696691 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.697774 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.697803 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.699588 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.699600 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.701653 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.701679 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.703220 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.703233 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.703599 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.704530 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.704556 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.707855 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.712322 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.712337 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.712376 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.713317 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.713823 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.713831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.713838 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.714236 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.714915 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.714924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.715338 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.715635 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.716233 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":407,"threads_started":1}
I20260619 18:41:35.740223 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.724958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:53105 (local address 127.3.85.190:37513)
0619 18:41:35.725119 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.725123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.725592 (+ 469us) server_negotiation.cc:407] Connection header received
0619 18:41:35.725851 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.725856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.725954 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.726068 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.727743 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.728760 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.730298 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.731010 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.731173 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.733212 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.733219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.733229 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.733261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.736685 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.737715 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.737722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.737725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.737906 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.738436 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.738466 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.738472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.739002 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.739181 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.739606 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.739859 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:41:35.740410 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.724969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.130:53105)
0619 18:41:35.725473 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.725489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.725685 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.726261 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.726266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.727006 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.727517 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.727540 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.728918 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.728924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.730119 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.730135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.731236 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.731243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.731425 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.732179 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.732196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.732978 (+ 782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.736854 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.736860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.736869 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.737494 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.738084 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.738092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.738098 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.738294 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.739146 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.739151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.739354 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.739511 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.739901 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260619 18:41:35.743674 11235 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:39885
I20260619 18:41:35.743701 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:39885 every 8 connection(s)
I20260619 18:41:35.744982 11235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:35.745456 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11235
I20260619 18:41:35.745653 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:35.746107 3414 external_mini_cluster.cc:1428] Setting key 60d9cd90587c42338de0e525b4cee58f
I20260619 18:41:35.747435 11371 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37513
I20260619 18:41:35.747694 11371 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:35.748440 11371 heartbeater.cc:511] Master 127.3.85.190:37513 requested a full tablet report, sending...
I20260619 18:41:35.750521 11023 ts_manager.cc:194] Registered new tserver with Master: 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885)
I20260619 18:41:35.751745 11023 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:45523
WARNING: no policy specified for oryx/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
I20260619 18:41:35.797968 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37513
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:36.140290 11375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:36.140616 11375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:36.140756 11375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:36.148367 11375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:36.148536 11375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:36.148697 11375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:41:36.148828 11375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:36.159174 11375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.85.190:37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:36.160729 11375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:36.162145 11375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:36.178517 11381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.179000 11383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.179070 11380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:36.180399 11375 server_base.cc:1034] running on GCE node
I20260619 18:41:36.181269 11375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:36.182602 11375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:36.183923 11375 hybrid_clock.cc:648] HybridClock initialized: now 1781894496183813 us; error 74 us; skew 500 ppm
I20260619 18:41:36.184334 11375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:41:36.187212 11231 heartbeater.cc:503] Master 127.3.85.190:37513 was elected leader, sending a full tablet report...
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894496, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:36.188663 11375 init.cc:377] Logged in from keytab as oryx/127.3.85.131@KRBTEST.COM (short username oryx)
I20260619 18:41:36.190662 11375 webserver.cc:460] Webserver started at http://127.3.85.131:40705/ using document root <none> and password file <none>
I20260619 18:41:36.191457 11375 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:36.191560 11375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:36.191920 11375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:36.195268 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "c0593978efd74090a12433bdf0896f27"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-09fp"
server_key: "5a0cd12ad1080f21a9054f31c959e370"
server_key_iv: "4ba40e2dab7d62e550731dcaa6f3c0ba"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.196144 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "c0593978efd74090a12433bdf0896f27"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-09fp"
server_key: "5a0cd12ad1080f21a9054f31c959e370"
server_key_iv: "4ba40e2dab7d62e550731dcaa6f3c0ba"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.201915 11375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:36.205968 11390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.207729 11375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:36.207937 11375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "c0593978efd74090a12433bdf0896f27"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-09fp"
server_key: "5a0cd12ad1080f21a9054f31c959e370"
server_key_iv: "4ba40e2dab7d62e550731dcaa6f3c0ba"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.208192 11375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:36.244313 11375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:36.249274 11375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:36.249600 11375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:36.252869 11375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:36.253001 11375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.253221 11375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:36.253327 11375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894496, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:36.282189 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.253315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:41255 (local address 127.3.85.190:37513)
0619 18:41:36.253807 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.253815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.255158 (+ 1343us) server_negotiation.cc:407] Connection header received
0619 18:41:36.257387 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.257396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.257499 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.257653 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.261467 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.262922 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.265589 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.266018 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.266159 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.271481 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.271490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.271499 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:36.271558 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.275911 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.277202 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.277207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.277209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.277398 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.278184 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.278189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.278193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.278566 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.278720 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.281521 (+ 2801us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.281793 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":235,"threads_started":1}
I20260619 18:41:36.282625 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.254073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.131:41255)
0619 18:41:36.254991 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.255030 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.256932 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.258060 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.258072 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.259262 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.260991 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.261038 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.263236 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.263248 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.265380 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.265399 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.266276 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.266283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.266590 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.267449 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:36.267474 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.271104 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.276123 (+ 5019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.276137 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.276172 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.276987 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.277559 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.277567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.277574 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.278005 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.278697 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.278716 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.279443 (+ 727us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.279903 (+ 460us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.280739 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":473,"threads_started":1}
I20260619 18:41:36.298228 11375 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:41503
I20260619 18:41:36.298238 11510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:41503 every 8 connection(s)
I20260619 18:41:36.299741 11375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:36.301211 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11375
I20260619 18:41:36.301386 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:36.301815 3414 external_mini_cluster.cc:1428] Setting key 7026fb00fb22250b832f651be373c95a
I20260619 18:41:36.301987 11511 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37513
I20260619 18:41:36.302769 11511 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:36.303567 11511 heartbeater.cc:511] Master 127.3.85.190:37513 requested a full tablet report, sending...
I20260619 18:41:36.305285 11024 ts_manager.cc:194] Registered new tserver with Master: c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:36.306267 11024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:41255
I20260619 18:41:36.308363 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.291010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:45721 (local address 127.3.85.190:37513)
0619 18:41:36.291209 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.291216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.292228 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 18:41:36.292449 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.292457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.292578 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.292712 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.294395 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.295686 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.297430 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.298189 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.298337 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.300517 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.300527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.300536 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:36.300602 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.305618 (+ 5016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.306276 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.306284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.306291 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.306499 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.306943 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.306948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.306952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.307327 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.307475 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.307941 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.308122 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:41:36.308667 11494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.291040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.3.85.131:45721)
0619 18:41:36.291996 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.292024 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.292300 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.292771 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.292778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.293610 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.294180 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.294198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.295854 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.295858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.297238 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.297252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.298413 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.298423 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.298669 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.299351 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:36.299370 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.300266 (+ 896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.305741 (+ 5475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.305745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.305748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.306159 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.306625 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.306632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.306635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.306798 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.307483 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.307489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.307691 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.307855 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.308212 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":523,"threads_started":1}
I20260619 18:41:36.319692 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:41:36.372391 11398 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:36.361471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.3.85.190:37513)
0619 18:41:36.361708 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.361716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.361872 (+ 156us) server_negotiation.cc:407] Connection header received
0619 18:41:36.362035 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.362039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.362105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.362214 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.363600 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.364323 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.365568 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.365986 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.366094 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.372027 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49310: BlockingRecv error: recv got EOF from 127.0.0.1:49310 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894496, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:36.390268 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.376097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.3.85.190:37513)
0619 18:41:36.376350 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.376356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.376590 (+ 234us) server_negotiation.cc:407] Connection header received
0619 18:41:36.376744 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.376748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.376812 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.376898 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.378054 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.378852 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.380062 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.380508 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.380612 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.384471 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.384476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.384480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:36.384519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.387247 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.388000 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.388004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.388007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.388139 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.388855 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.388859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.388864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.389169 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.389385 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.389728 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.389907 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:41:36.394750 11024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:36.401885 11024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:36.429486 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.415960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.3.85.131:41503)
0619 18:41:36.416666 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.416675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.416708 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:41:36.416799 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.416806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.417081 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.417368 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.418253 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.420668 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.425941 (+ 5273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.427683 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.427852 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.428117 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.428859 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.429072 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":1260,"threads_started":1}
I20260619 18:41:36.430305 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.415314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:41503 (local address 127.0.0.1:55908)
0619 18:41:36.416028 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.416059 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.416282 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.417362 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.417370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.417403 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.417955 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.417973 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.420901 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.420909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.425605 (+ 4696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.425636 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.427924 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.427931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.428215 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.428270 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.428507 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.429978 (+ 1471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":873,"threads_started":1}
I20260619 18:41:36.435051 11441 tablet_service.cc:1467] Processing CreateTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (DEFAULT_TABLE table=test-table [id=43dae048f3dc41da839764452e68abe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.436316 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.421802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.3.85.130:39885)
0619 18:41:36.427107 (+ 5305us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.427121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.427165 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:41:36.427302 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.427313 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.427711 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.427993 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.429311 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.430908 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.433610 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.435122 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.435281 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.435532 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.435822 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.436011 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5057,"thread_start_us":4885,"threads_started":1}
I20260619 18:41:36.436484 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.417051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45125 (local address 127.0.0.1:45566)
0619 18:41:36.421517 (+ 4466us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.421549 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.421682 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.429206 (+ 7524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.429210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.429236 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.429617 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.429633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.431493 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.431498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.433697 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.433711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.435567 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.435574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.435758 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.435789 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.435856 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.435990 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4271,"mutex_wait_us":7,"thread_start_us":4046,"threads_started":1}
I20260619 18:41:36.436513 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.421748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39885 (local address 127.0.0.1:55270)
0619 18:41:36.422539 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.422564 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.422714 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.428598 (+ 5884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.428604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.428638 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.429092 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.429106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.431062 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.431066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.433365 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.433384 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.435344 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.435351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.435582 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.435624 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.435750 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.435931 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":95,"thread_start_us":381,"threads_started":1}
I20260619 18:41:36.437335 11441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.437577 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.417316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.3.85.129:45125)
0619 18:41:36.428599 (+ 11283us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.428609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.428634 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:41:36.428724 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.428730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.428977 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.429836 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.429996 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.431322 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.433883 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.435313 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.436936 (+ 1623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.437103 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.437161 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.437280 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11138,"thread_start_us":11006,"threads_started":1}
I20260619 18:41:36.440454 11161 tablet_service.cc:1467] Processing CreateTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (DEFAULT_TABLE table=test-table [id=43dae048f3dc41da839764452e68abe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.440603 11301 tablet_service.cc:1467] Processing CreateTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (DEFAULT_TABLE table=test-table [id=43dae048f3dc41da839764452e68abe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.441753 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.442620 11301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.451488 11535 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Bootstrap starting.
I20260619 18:41:36.455440 11535 tablet_bootstrap.cc:654] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.456648 11535 log.cc:824] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.457767 11536 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Bootstrap starting.
I20260619 18:41:36.458812 11537 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Bootstrap starting.
I20260619 18:41:36.462596 11536 tablet_bootstrap.cc:654] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.464022 11537 tablet_bootstrap.cc:654] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.464035 11536 log.cc:824] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.465421 11537 log.cc:824] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.466069 11535 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: No bootstrap required, opened a new log
I20260619 18:41:36.466507 11535 ts_tablet_manager.cc:1397] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260619 18:41:36.466979 11536 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: No bootstrap required, opened a new log
I20260619 18:41:36.467252 11536 ts_tablet_manager.cc:1397] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:41:36.472807 11537 tablet_bootstrap.cc:492] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: No bootstrap required, opened a new log
I20260619 18:41:36.473261 11537 ts_tablet_manager.cc:1397] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260619 18:41:36.477577 11536 raft_consensus.cc:348] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.478276 11536 raft_consensus.cc:374] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.478473 11536 raft_consensus.cc:729] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.478533 11535 raft_consensus.cc:348] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.479136 11535 raft_consensus.cc:374] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.479287 11535 raft_consensus.cc:729] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0593978efd74090a12433bdf0896f27, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.479269 11536 consensus_queue.cc:260] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.480082 11535 consensus_queue.cc:260] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.488724 11535 ts_tablet_manager.cc:1428] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260619 18:41:36.489259 11511 heartbeater.cc:503] Master 127.3.85.190:37513 was elected leader, sending a full tablet report...
I20260619 18:41:36.488960 11537 raft_consensus.cc:348] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.489579 11537 raft_consensus.cc:374] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.489692 11537 raft_consensus.cc:729] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.490587 11537 consensus_queue.cc:260] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.492313 11536 ts_tablet_manager.cc:1428] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Time spent starting tablet: real 0.025s user 0.017s sys 0.008s
I20260619 18:41:36.493677 11371 heartbeater.cc:503] Master 127.3.85.190:37513 was elected leader, sending a full tablet report...
I20260619 18:41:36.494321 11537 ts_tablet_manager.cc:1428] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
W20260619 18:41:36.497874 11372 tablet.cc:2369] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:36.551499 11512 tablet.cc:2369] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:36.572582 11541 raft_consensus.cc:482] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:36.572896 11541 raft_consensus.cc:504] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.575037 11541 leader_election.cc:288] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503), 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885)
I20260619 18:41:36.581523 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.575355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:46957 (local address 127.3.85.131:41503)
0619 18:41:36.575575 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.575584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.576127 (+ 543us) server_negotiation.cc:407] Connection header received
0619 18:41:36.576383 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.576391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.576491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.576587 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.577201 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.577907 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.579806 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.580580 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.580782 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.580861 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.581141 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.581265 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:41:36.581611 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.575375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:41503 (local address 127.3.85.129:46957)
0619 18:41:36.576006 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.576022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.576199 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.576665 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.576669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.576694 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.577056 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.577065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.578046 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.578054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.579633 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.579645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.580797 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.580803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.580963 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.580987 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.581104 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.581242 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":349,"threads_started":1}
I20260619 18:41:36.581797 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.576424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:41833 (local address 127.3.85.130:39885)
0619 18:41:36.576632 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.576637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.576858 (+ 221us) server_negotiation.cc:407] Connection header received
0619 18:41:36.577062 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.577067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.577128 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.577218 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.577697 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.578379 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.580116 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.580863 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.581057 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.581134 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.581392 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.581526 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:41:36.582582 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.576169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39885 (local address 127.3.85.129:41833)
0619 18:41:36.576760 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.576773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.576926 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.577240 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.577244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.577266 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.577555 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.577564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.578570 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.578575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.579947 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.579966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.581086 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.581092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.581239 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.581261 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.581367 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.581500 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":263,"threads_started":1}
I20260619 18:41:36.582872 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b7deaad9ed4aa1a8bbd16df2289b2f" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27" is_pre_election: true
I20260619 18:41:36.583441 11461 raft_consensus.cc:2454] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.583684 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b7deaad9ed4aa1a8bbd16df2289b2f" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9" is_pre_election: true
I20260619 18:41:36.584120 11111 leader_election.cc:302] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, c0593978efd74090a12433bdf0896f27; no voters:
I20260619 18:41:36.584174 11321 raft_consensus.cc:2454] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.584609 11541 raft_consensus.cc:2781] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:36.584777 11541 raft_consensus.cc:482] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:36.584898 11541 raft_consensus.cc:3037] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.587301 11541 raft_consensus.cc:504] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.587970 11541 leader_election.cc:288] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Requested vote from peers c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503), 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885)
I20260619 18:41:36.588560 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b7deaad9ed4aa1a8bbd16df2289b2f" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27"
I20260619 18:41:36.588596 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b7deaad9ed4aa1a8bbd16df2289b2f" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9"
I20260619 18:41:36.588778 11461 raft_consensus.cc:3037] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.588789 11321 raft_consensus.cc:3037] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.591156 11321 raft_consensus.cc:2454] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.591168 11461 raft_consensus.cc:2454] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.591665 11111 leader_election.cc:302] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:36.591975 11541 raft_consensus.cc:2781] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:36.592504 11541 raft_consensus.cc:686] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 1 LEADER]: Becoming Leader. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Running, Role: LEADER
I20260619 18:41:36.593065 11541 consensus_queue.cc:237] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } }
I20260619 18:41:36.598176 11022 catalog_manager.cc:5495] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:36.624603 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.618763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.3.85.129:45125)
0619 18:41:36.619001 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.619010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.619142 (+ 132us) server_negotiation.cc:407] Connection header received
0619 18:41:36.619473 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.619484 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.619627 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.619787 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:36.620177 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.621014 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.622365 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.622853 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.622984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.623426 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:36.623999 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:36.624178 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.624237 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.624322 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:41:36.633472 11321 raft_consensus.cc:1261] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.633472 11461 raft_consensus.cc:1261] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.634779 11546 consensus_queue.cc:1035] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.635226 11541 consensus_queue.cc:1035] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.665315 11024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:36.665627 11024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:36.669504 11024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Sending DeleteTablet for 3 replicas of tablet f6b7deaad9ed4aa1a8bbd16df2289b2f
I20260619 18:41:36.670845 11441 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55908
I20260619 18:41:36.671022 11301 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55270
I20260619 18:41:36.672173 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b7deaad9ed4aa1a8bbd16df2289b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45566
I20260619 18:41:36.672463 11562 tablet_replica.cc:330] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: stopping tablet replica
I20260619 18:41:36.673044 11562 raft_consensus.cc:2229] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:36.673115 11022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:41:36.673770 11562 raft_consensus.cc:2258] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:41:36.673883 11022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:36.676637 11562 ts_tablet_manager.cc:1905] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:36.683322 11440 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 (DEFAULT_TABLE table=test-table [id=3be72ef7c5d64fe490c2d0a83beba113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.683966 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab0daa0ab7454ba02c3d59e8b70066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.685412 11561 tablet_replica.cc:330] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: stopping tablet replica
I20260619 18:41:36.685982 11561 raft_consensus.cc:2229] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:36.687455 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 (DEFAULT_TABLE table=test-table [id=3be72ef7c5d64fe490c2d0a83beba113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.687621 11301 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 (DEFAULT_TABLE table=test-table [id=3be72ef7c5d64fe490c2d0a83beba113]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.688051 11301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab0daa0ab7454ba02c3d59e8b70066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.688081 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab0daa0ab7454ba02c3d59e8b70066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.688915 11561 raft_consensus.cc:2258] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:36.695081 11537 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Bootstrap starting.
I20260619 18:41:36.695607 11561 ts_tablet_manager.cc:1905] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:36.696260 11563 tablet_replica.cc:330] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: stopping tablet replica
I20260619 18:41:36.696770 11537 tablet_bootstrap.cc:654] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.696943 11563 raft_consensus.cc:2229] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:36.697662 11563 raft_consensus.cc:2258] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:36.699852 11562 ts_tablet_manager.cc:1918] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:36.700110 11562 log.cc:1196] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/f6b7deaad9ed4aa1a8bbd16df2289b2f
I20260619 18:41:36.700256 11536 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Bootstrap starting.
I20260619 18:41:36.700891 11562 ts_tablet_manager.cc:1939] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 3a6f73c544e848be83a8527106fcfed9: Deleting consensus metadata
I20260619 18:41:36.702248 11536 tablet_bootstrap.cc:654] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.704130 11008 catalog_manager.cc:4841] TS 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885): tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (table test-table [id=43dae048f3dc41da839764452e68abe7]) successfully deleted
I20260619 18:41:36.708958 11561 ts_tablet_manager.cc:1918] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:36.709227 11561 log.cc:1196] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/f6b7deaad9ed4aa1a8bbd16df2289b2f
I20260619 18:41:36.709846 11561 ts_tablet_manager.cc:1939] T f6b7deaad9ed4aa1a8bbd16df2289b2f P c0593978efd74090a12433bdf0896f27: Deleting consensus metadata
I20260619 18:41:36.714340 11008 catalog_manager.cc:4841] TS c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503): tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (table test-table [id=43dae048f3dc41da839764452e68abe7]) successfully deleted
I20260619 18:41:36.715580 11535 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Bootstrap starting.
I20260619 18:41:36.717406 11535 tablet_bootstrap.cc:654] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.719699 11563 ts_tablet_manager.cc:1905] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:36.728621 11536 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: No bootstrap required, opened a new log
I20260619 18:41:36.728826 11536 ts_tablet_manager.cc:1397] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
I20260619 18:41:36.729285 11537 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: No bootstrap required, opened a new log
I20260619 18:41:36.729454 11537 ts_tablet_manager.cc:1397] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.021s
I20260619 18:41:36.729809 11536 raft_consensus.cc:348] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.730074 11536 raft_consensus.cc:374] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.730194 11536 raft_consensus.cc:729] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.730301 11537 raft_consensus.cc:348] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.730600 11537 raft_consensus.cc:374] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.730692 11537 raft_consensus.cc:729] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.730968 11537 consensus_queue.cc:260] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.731640 11537 ts_tablet_manager.cc:1428] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:36.731966 11536 consensus_queue.cc:260] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.732661 11536 ts_tablet_manager.cc:1428] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 18:41:36.733966 11563 ts_tablet_manager.cc:1918] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:36.734198 11563 log.cc:1196] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/f6b7deaad9ed4aa1a8bbd16df2289b2f
I20260619 18:41:36.734916 11563 ts_tablet_manager.cc:1939] T f6b7deaad9ed4aa1a8bbd16df2289b2f P 36ef2e84b2204305a21f1b292b0fd930: Deleting consensus metadata
I20260619 18:41:36.737219 11008 catalog_manager.cc:4841] TS 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125): tablet f6b7deaad9ed4aa1a8bbd16df2289b2f (table test-table [id=43dae048f3dc41da839764452e68abe7]) successfully deleted
I20260619 18:41:36.737890 11535 tablet_bootstrap.cc:492] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: No bootstrap required, opened a new log
I20260619 18:41:36.738065 11535 ts_tablet_manager.cc:1397] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.007s
I20260619 18:41:36.738832 11535 raft_consensus.cc:348] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.739010 11535 raft_consensus.cc:374] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.739120 11535 raft_consensus.cc:729] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0593978efd74090a12433bdf0896f27, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.739399 11535 consensus_queue.cc:260] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.739945 11535 ts_tablet_manager.cc:1428] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:36.812340 11558 raft_consensus.cc:482] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:36.812731 11558 raft_consensus.cc:504] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.813690 11558 leader_election.cc:288] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:36.814419 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab0daa0ab7454ba02c3d59e8b70066" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27" is_pre_election: true
I20260619 18:41:36.814436 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab0daa0ab7454ba02c3d59e8b70066" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9" is_pre_election: true
I20260619 18:41:36.814718 11461 raft_consensus.cc:2454] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.814772 11321 raft_consensus.cc:2454] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.815209 11111 leader_election.cc:302] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, c0593978efd74090a12433bdf0896f27; no voters:
I20260619 18:41:36.815567 11558 raft_consensus.cc:2781] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:36.815711 11558 raft_consensus.cc:482] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:36.815848 11558 raft_consensus.cc:3037] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.819592 11558 raft_consensus.cc:504] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.820472 11558 leader_election.cc:288] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Requested vote from peers 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:36.821168 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab0daa0ab7454ba02c3d59e8b70066" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27"
I20260619 18:41:36.821164 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab0daa0ab7454ba02c3d59e8b70066" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9"
I20260619 18:41:36.821380 11461 raft_consensus.cc:3037] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.821437 11321 raft_consensus.cc:3037] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.823832 11321 raft_consensus.cc:2454] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.824395 11111 leader_election.cc:302] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:36.824416 11461 raft_consensus.cc:2454] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.824837 11558 raft_consensus.cc:2781] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:36.825014 11558 raft_consensus.cc:686] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 LEADER]: Becoming Leader. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Running, Role: LEADER
I20260619 18:41:36.825333 11558 consensus_queue.cc:237] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.828440 11022 catalog_manager.cc:5495] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:41:36 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894494, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.190@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:36.899289 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.877158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.3.85.190:37513)
0619 18:41:36.877372 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.877379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.878095 (+ 716us) server_negotiation.cc:407] Connection header received
0619 18:41:36.878241 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.878247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.878329 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.878477 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.879717 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.880877 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.882184 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.882878 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.883021 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.893327 (+ 10306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.893333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.893337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:36.893374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.896204 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.897293 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.897297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.897300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.897433 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:36.897845 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:36.897848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:36.897850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:36.898118 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.898230 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.898793 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.898989 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:41:36.899798 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.877149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37513 (local address 127.0.0.1:49338)
0619 18:41:36.877955 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.877979 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.878118 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.878537 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.878541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.879215 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.879526 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.879536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.881037 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.881040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.882010 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.882022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.883095 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.883100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.883218 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.890073 (+ 6855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:36.890101 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.893062 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.896403 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.896421 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.896430 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.897109 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.897547 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:36.897552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:36.897555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:36.897732 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:36.898253 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.898265 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.898502 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.898710 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.899201 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":138,"thread_start_us":477,"threads_started":1}
I20260619 18:41:36.905398 11022 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:49338:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:41:36.906642 11022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:36.914625 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 71a2a2eafb784aae926de12cab105af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3096af00b3476e8fc0818ab608f2cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:41:36.914820 11301 tablet_service.cc:1467] Processing CreateTablet for tablet 71a2a2eafb784aae926de12cab105af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3096af00b3476e8fc0818ab608f2cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:41:36.915052 11440 tablet_service.cc:1467] Processing CreateTablet for tablet 71a2a2eafb784aae926de12cab105af4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3096af00b3476e8fc0818ab608f2cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:41:36.915277 11301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2a2eafb784aae926de12cab105af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.915270 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2a2eafb784aae926de12cab105af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.915570 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a2a2eafb784aae926de12cab105af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.923540 11537 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: Bootstrap starting.
I20260619 18:41:36.923547 11536 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: Bootstrap starting.
I20260619 18:41:36.925463 11535 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: Bootstrap starting.
I20260619 18:41:36.925546 11537 tablet_bootstrap.cc:654] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.925562 11536 tablet_bootstrap.cc:654] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.927510 11535 tablet_bootstrap.cc:654] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: Neither blocks nor log segments found. Creating new log.
W20260619 18:41:36.929033 11232 tablet.cc:2369] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:36.935231 11535 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: No bootstrap required, opened a new log
I20260619 18:41:36.935462 11535 ts_tablet_manager.cc:1397] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 18:41:36.935626 11537 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: No bootstrap required, opened a new log
I20260619 18:41:36.935820 11537 ts_tablet_manager.cc:1397] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260619 18:41:36.936388 11535 raft_consensus.cc:348] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.936672 11535 raft_consensus.cc:374] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.936757 11535 raft_consensus.cc:729] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0593978efd74090a12433bdf0896f27, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.937294 11535 consensus_queue.cc:260] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.937449 11537 raft_consensus.cc:348] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.937776 11537 raft_consensus.cc:374] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.937884 11537 raft_consensus.cc:729] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.937883 11542 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:36.938266 11542 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:36.938184 11537 consensus_queue.cc:260] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.938964 11537 ts_tablet_manager.cc:1428] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:36.938818 11543 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:36.939217 11543 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:36.939883 11535 ts_tablet_manager.cc:1428] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 18:41:36.942905 11536 tablet_bootstrap.cc:492] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: No bootstrap required, opened a new log
I20260619 18:41:36.943073 11536 ts_tablet_manager.cc:1397] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20260619 18:41:36.943738 11536 raft_consensus.cc:348] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.944027 11536 raft_consensus.cc:374] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.944131 11536 raft_consensus.cc:729] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.944511 11536 consensus_queue.cc:260] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.945626 11536 ts_tablet_manager.cc:1428] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:36.945003 11558 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:36.946040 11558 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:36.956749 11558 raft_consensus.cc:482] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:36.957005 11558 raft_consensus.cc:504] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.957859 11558 leader_election.cc:288] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:36.958475 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2a2eafb784aae926de12cab105af4" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9" is_pre_election: true
I20260619 18:41:36.958645 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2a2eafb784aae926de12cab105af4" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27" is_pre_election: true
I20260619 18:41:36.958779 11321 raft_consensus.cc:2454] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.958899 11461 raft_consensus.cc:2454] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 0.
I20260619 18:41:36.959304 11111 leader_election.cc:302] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:36.959640 11558 raft_consensus.cc:2781] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:36.959764 11558 raft_consensus.cc:482] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:36.959913 11558 raft_consensus.cc:3037] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.963295 11558 raft_consensus.cc:504] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.964128 11558 leader_election.cc:288] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Requested vote from peers 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:36.964668 11321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2a2eafb784aae926de12cab105af4" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6f73c544e848be83a8527106fcfed9"
I20260619 18:41:36.964931 11321 raft_consensus.cc:3037] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.964937 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a2a2eafb784aae926de12cab105af4" candidate_uuid: "36ef2e84b2204305a21f1b292b0fd930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27"
I20260619 18:41:36.965220 11461 raft_consensus.cc:3037] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.967535 11461 raft_consensus.cc:2454] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.967947 11321 raft_consensus.cc:2454] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36ef2e84b2204305a21f1b292b0fd930 in term 1.
I20260619 18:41:36.968179 11111 leader_election.cc:302] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, c0593978efd74090a12433bdf0896f27; no voters:
I20260619 18:41:36.968518 11558 raft_consensus.cc:2781] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:36.968719 11558 raft_consensus.cc:686] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 LEADER]: Becoming Leader. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Running, Role: LEADER
I20260619 18:41:36.968994 11558 consensus_queue.cc:237] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:36.970919 11555 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: TxnStatusTablet state changed. Reason: New leader 36ef2e84b2204305a21f1b292b0fd930. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:36.971371 11555 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:41:36.972142 11578 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:41:36.972378 11578 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:41:36.972990 11023 catalog_manager.cc:5495] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:36.989924 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.984406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.3.85.129:45125)
0619 18:41:36.984616 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.984623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.984700 (+ 77us) server_negotiation.cc:407] Connection header received
0619 18:41:36.985031 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.985046 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.985135 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.985243 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:36.985643 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.986657 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.987983 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.988453 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.988606 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.989107 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:36.989231 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:36.989340 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.989494 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.989582 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 18:41:36.989924 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.984319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45125 (local address 127.0.0.1:45584)
0619 18:41:36.984593 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.984615 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.984821 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.985270 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.985274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.985287 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:41:36.985476 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.985484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.986794 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.986803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.987812 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.987825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.988632 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.988638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.988833 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.988897 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:41:36.989343 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:41:36.989366 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.989471 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.989644 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:41:36.994052 11321 raft_consensus.cc:1261] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.994221 11461 raft_consensus.cc:1261] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.994863 11558 consensus_queue.cc:1035] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.995270 11555 consensus_queue.cc:1035] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.998816 11542 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: TxnStatusTablet state changed. Reason: New leader 36ef2e84b2204305a21f1b292b0fd930. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:36.999050 11542 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:37.001209 11543 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: TxnStatusTablet state changed. Reason: New leader 36ef2e84b2204305a21f1b292b0fd930. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:37.001516 11543 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:37.002714 11558 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:37.003091 11558 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:41:37.003451 11549 mvcc.cc:205] Tried to move back new op lower bound from 7298639859681243136 to 7298639859588030464. Current Snapshot: MvccSnapshot[applied={T|T < 7298639859681243136 or (T in {7298639859681243136})}]
I20260619 18:41:37.004281 11542 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:37.004524 11542 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P c0593978efd74090a12433bdf0896f27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:37.007414 11541 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:37.007732 11541 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 36ef2e84b2204305a21f1b292b0fd930: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:41:37.012113 11543 tablet_replica.cc:439] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36ef2e84b2204305a21f1b292b0fd930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } } }
I20260619 18:41:37.012388 11543 tablet_replica.cc:442] T 71a2a2eafb784aae926de12cab105af4 P 3a6f73c544e848be83a8527106fcfed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:41:37.030085 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.023403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45125 (local address 127.3.85.129:53845)
0619 18:41:37.024213 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:37.024231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:37.024359 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:37.024707 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:37.024710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:37.024719 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:41:37.025043 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.025052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.026499 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.026504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.027822 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.027836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.028907 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.028912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.029038 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.029097 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.029542 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.029553 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:41:37.029621 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:41:37.029727 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":445,"threads_started":1}
I20260619 18:41:37.030109 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.023395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:53845 (local address 127.3.85.129:45125)
0619 18:41:37.023622 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.023629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.024333 (+ 704us) server_negotiation.cc:407] Connection header received
0619 18:41:37.024492 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.024498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.024583 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.024691 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:37.025191 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.026313 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.027993 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.028707 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.028851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.029278 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.029438 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.029540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.029681 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.029762 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:41:37.039938 11321 raft_consensus.cc:1261] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.040050 11461 raft_consensus.cc:1261] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Refusing update from remote peer 36ef2e84b2204305a21f1b292b0fd930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.040549 11558 consensus_queue.cc:1035] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.040863 11555 consensus_queue.cc:1035] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.106130 11023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.106352 11023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.109386 11023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Sending DeleteTablet for 3 replicas of tablet 56ab0daa0ab7454ba02c3d59e8b70066
I20260619 18:41:37.110338 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45566
I20260619 18:41:37.110531 11440 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55908
I20260619 18:41:37.110705 11301 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab0daa0ab7454ba02c3d59e8b70066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55270
I20260619 18:41:37.111030 11562 tablet_replica.cc:330] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: stopping tablet replica
I20260619 18:41:37.111035 11563 tablet_replica.cc:330] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: stopping tablet replica
I20260619 18:41:37.111119 11561 tablet_replica.cc:330] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: stopping tablet replica
I20260619 18:41:37.111243 11562 raft_consensus.cc:2229] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.111245 11563 raft_consensus.cc:2229] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:37.111310 11561 raft_consensus.cc:2229] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.111436 11562 raft_consensus.cc:2258] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.111505 11561 raft_consensus.cc:2258] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.111603 11563 raft_consensus.cc:2258] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.113251 11561 ts_tablet_manager.cc:1905] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.113273 11563 ts_tablet_manager.cc:1905] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.116991 11562 ts_tablet_manager.cc:1905] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.120226 11563 ts_tablet_manager.cc:1918] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.120474 11563 log.cc:1196] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/56ab0daa0ab7454ba02c3d59e8b70066
I20260619 18:41:37.120988 11563 ts_tablet_manager.cc:1939] T 56ab0daa0ab7454ba02c3d59e8b70066 P 36ef2e84b2204305a21f1b292b0fd930: Deleting consensus metadata
I20260619 18:41:37.122277 11008 catalog_manager.cc:4841] TS 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125): tablet 56ab0daa0ab7454ba02c3d59e8b70066 (table test-table [id=3be72ef7c5d64fe490c2d0a83beba113]) successfully deleted
I20260619 18:41:37.123114 11561 ts_tablet_manager.cc:1918] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.123273 11561 log.cc:1196] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/56ab0daa0ab7454ba02c3d59e8b70066
I20260619 18:41:37.123787 11561 ts_tablet_manager.cc:1939] T 56ab0daa0ab7454ba02c3d59e8b70066 P c0593978efd74090a12433bdf0896f27: Deleting consensus metadata
I20260619 18:41:37.124744 11562 ts_tablet_manager.cc:1918] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.124923 11562 log.cc:1196] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/56ab0daa0ab7454ba02c3d59e8b70066
I20260619 18:41:37.124997 11008 catalog_manager.cc:4841] TS c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503): tablet 56ab0daa0ab7454ba02c3d59e8b70066 (table test-table [id=3be72ef7c5d64fe490c2d0a83beba113]) successfully deleted
I20260619 18:41:37.125407 11562 ts_tablet_manager.cc:1939] T 56ab0daa0ab7454ba02c3d59e8b70066 P 3a6f73c544e848be83a8527106fcfed9: Deleting consensus metadata
I20260619 18:41:37.126338 11008 catalog_manager.cc:4841] TS 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885): tablet 56ab0daa0ab7454ba02c3d59e8b70066 (table test-table [id=3be72ef7c5d64fe490c2d0a83beba113]) successfully deleted
Jun 19 18:41:37 dist-test-slave-09fp krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:41:37 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:37 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:41:37.184957 11398 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:37.169442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.3.85.190:37513)
0619 18:41:37.169716 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.169728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.170366 (+ 638us) server_negotiation.cc:407] Connection header received
0619 18:41:37.170661 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.170668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.170795 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.170923 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:37.173082 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.174364 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.176567 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.177325 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.177455 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.184696 (+ 7241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49352: BlockingRecv error: recv got EOF from 127.0.0.1:49352 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 18:41:37 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM
I20260619 18:41:37.212743 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.192628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.3.85.190:37513)
0619 18:41:37.192875 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.192882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.193515 (+ 633us) server_negotiation.cc:407] Connection header received
0619 18:41:37.193727 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.193735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.193841 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.193954 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:37.196051 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.197217 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.199273 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.199971 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.200107 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.205464 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:37.205472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:37.205477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:37.205525 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:37.209191 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:37.210198 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:37.210204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:37.210208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:37.210355 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:37.210802 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:37.210808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:37.210811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:37.211158 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:37.211436 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.212082 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.212332 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:41:37.216464 11023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:37.217175 11023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:37.224607 11161 tablet_service.cc:1467] Processing CreateTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 (DEFAULT_TABLE table=test-table [id=659e2b392f2d4f058783b05f40eac9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.224826 11301 tablet_service.cc:1467] Processing CreateTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 (DEFAULT_TABLE table=test-table [id=659e2b392f2d4f058783b05f40eac9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.225124 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442dab63af247d8a842d4e3c2e6b4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.225263 11301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442dab63af247d8a842d4e3c2e6b4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.230079 11537 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Bootstrap starting.
I20260619 18:41:37.230872 11440 tablet_service.cc:1467] Processing CreateTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 (DEFAULT_TABLE table=test-table [id=659e2b392f2d4f058783b05f40eac9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.231312 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442dab63af247d8a842d4e3c2e6b4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.231722 11537 tablet_bootstrap.cc:654] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.232671 11536 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Bootstrap starting.
I20260619 18:41:37.234436 11536 tablet_bootstrap.cc:654] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.240383 11535 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Bootstrap starting.
I20260619 18:41:37.242730 11535 tablet_bootstrap.cc:654] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.243991 11537 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: No bootstrap required, opened a new log
I20260619 18:41:37.244273 11537 ts_tablet_manager.cc:1397] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260619 18:41:37.245404 11537 raft_consensus.cc:348] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.245957 11537 raft_consensus.cc:374] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.246124 11537 raft_consensus.cc:729] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.246441 11537 consensus_queue.cc:260] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.248718 11537 ts_tablet_manager.cc:1428] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:41:37.253284 11536 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: No bootstrap required, opened a new log
I20260619 18:41:37.253496 11536 ts_tablet_manager.cc:1397] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.002s
I20260619 18:41:37.254105 11535 tablet_bootstrap.cc:492] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: No bootstrap required, opened a new log
I20260619 18:41:37.254271 11535 ts_tablet_manager.cc:1397] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260619 18:41:37.254386 11536 raft_consensus.cc:348] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.254742 11536 raft_consensus.cc:374] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.254846 11536 raft_consensus.cc:729] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.255146 11536 consensus_queue.cc:260] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.255141 11535 raft_consensus.cc:348] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.255398 11535 raft_consensus.cc:374] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.255494 11535 raft_consensus.cc:729] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0593978efd74090a12433bdf0896f27, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.255753 11536 ts_tablet_manager.cc:1428] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:37.255740 11535 consensus_queue.cc:260] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.256304 11535 ts_tablet_manager.cc:1428] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:41:37.340737 11543 raft_consensus.cc:482] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:37.340988 11543 raft_consensus.cc:504] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.342850 11543 leader_election.cc:288] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:37.353940 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.343149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:38915 (local address 127.3.85.129:45125)
0619 18:41:37.343350 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.343356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.343761 (+ 405us) server_negotiation.cc:407] Connection header received
0619 18:41:37.344088 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.344097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.344200 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.344277 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:37.345245 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.346782 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.350305 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.352191 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.352381 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.352517 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.353247 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.353476 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:41:37.355365 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.343097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45125 (local address 127.3.85.130:38915)
0619 18:41:37.343667 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:37.343687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:37.343868 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:37.344553 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:37.344561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:37.344594 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:37.345029 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.345042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.347040 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.347049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.350029 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.350055 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.352713 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.352722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.352991 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.353046 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:41:37.354601 (+ 1555us) client_negotiation.cc:240] Negotiation successful
0619 18:41:37.354851 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":290,"threads_started":1}
I20260619 18:41:37.357575 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442dab63af247d8a842d4e3c2e6b4c8" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef2e84b2204305a21f1b292b0fd930" is_pre_election: true
I20260619 18:41:37.358088 11181 raft_consensus.cc:2454] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 0.
I20260619 18:41:37.358888 11251 leader_election.cc:302] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:37.359412 11543 raft_consensus.cc:2781] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:37.359556 11543 raft_consensus.cc:482] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:37.359669 11543 raft_consensus.cc:3037] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.363322 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.343960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:52329 (local address 127.3.85.131:41503)
0619 18:41:37.357111 (+ 13151us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.357117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.357143 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:37.357214 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.357219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.357353 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.357488 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:37.358056 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.359231 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.361260 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.362202 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.362422 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.362539 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.362828 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.362972 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12989,"thread_start_us":12827,"threads_started":1}
I20260619 18:41:37.363309 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.343809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:41503 (local address 127.3.85.130:52329)
0619 18:41:37.355910 (+ 12101us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:37.355938 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:37.356095 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:37.357506 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:37.357511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:37.357541 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:37.357743 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.357753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.359413 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.359418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.361073 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.361089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.362414 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.362420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.362656 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.362682 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:41:37.362803 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:41:37.362973 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11951,"thread_start_us":12584,"threads_started":1}
I20260619 18:41:37.363190 11543 raft_consensus.cc:504] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.363950 11543 leader_election.cc:288] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 election: Requested vote from peers 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:37.364508 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442dab63af247d8a842d4e3c2e6b4c8" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef2e84b2204305a21f1b292b0fd930"
I20260619 18:41:37.364600 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442dab63af247d8a842d4e3c2e6b4c8" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27" is_pre_election: true
I20260619 18:41:37.364770 11181 raft_consensus.cc:3037] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.364738 11460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442dab63af247d8a842d4e3c2e6b4c8" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27"
I20260619 18:41:37.364846 11461 raft_consensus.cc:2454] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 0.
I20260619 18:41:37.365007 11460 raft_consensus.cc:3037] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.367282 11181 raft_consensus.cc:2454] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 1.
I20260619 18:41:37.367905 11251 leader_election.cc:302] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:37.368038 11460 raft_consensus.cc:2454] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 1.
I20260619 18:41:37.368352 11543 raft_consensus.cc:2781] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:37.369356 11543 raft_consensus.cc:686] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Running, Role: LEADER
I20260619 18:41:37.369808 11543 consensus_queue.cc:237] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.373946 11023 catalog_manager.cc:5495] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "3a6f73c544e848be83a8527106fcfed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:37.410866 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.404016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.3.85.130:39885)
0619 18:41:37.404609 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.404616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.404642 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:37.404753 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.404759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.404866 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.405068 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:37.405408 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.406643 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.407966 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.408662 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.408803 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.409220 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.410032 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.410228 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.410353 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.410493 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260619 18:41:37.415071 11460 raft_consensus.cc:1261] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Refusing update from remote peer 3a6f73c544e848be83a8527106fcfed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.415246 11181 raft_consensus.cc:1261] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Refusing update from remote peer 3a6f73c544e848be83a8527106fcfed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.415935 11606 consensus_queue.cc:1035] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.416360 11543 consensus_queue.cc:1035] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.438817 11022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.439109 11022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.442901 11022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Sending DeleteTablet for 3 replicas of tablet 7442dab63af247d8a842d4e3c2e6b4c8
I20260619 18:41:37.444552 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45566
I20260619 18:41:37.444922 11301 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55270
I20260619 18:41:37.445027 11024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:41:37.445226 11440 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442dab63af247d8a842d4e3c2e6b4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55908
I20260619 18:41:37.445544 11563 tablet_replica.cc:330] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: stopping tablet replica
W20260619 18:41:37.445712 11024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:37.445736 11563 raft_consensus.cc:2229] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.445983 11563 raft_consensus.cc:2258] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.446295 11562 tablet_replica.cc:330] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: stopping tablet replica
I20260619 18:41:37.446512 11562 raft_consensus.cc:2229] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:37.447057 11561 tablet_replica.cc:330] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: stopping tablet replica
I20260619 18:41:37.447091 11562 raft_consensus.cc:2258] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.447237 11561 raft_consensus.cc:2229] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.447461 11561 raft_consensus.cc:2258] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.448562 11563 ts_tablet_manager.cc:1905] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.448783 11561 ts_tablet_manager.cc:1905] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.451793 11562 ts_tablet_manager.cc:1905] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.456828 11563 ts_tablet_manager.cc:1918] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:37.457100 11563 log.cc:1196] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/7442dab63af247d8a842d4e3c2e6b4c8
I20260619 18:41:37.457629 11563 ts_tablet_manager.cc:1939] T 7442dab63af247d8a842d4e3c2e6b4c8 P 36ef2e84b2204305a21f1b292b0fd930: Deleting consensus metadata
I20260619 18:41:37.459230 11008 catalog_manager.cc:4841] TS 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125): tablet 7442dab63af247d8a842d4e3c2e6b4c8 (table test-table [id=659e2b392f2d4f058783b05f40eac9d9]) successfully deleted
I20260619 18:41:37.462965 11562 ts_tablet_manager.cc:1918] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:37.462955 11561 ts_tablet_manager.cc:1918] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:37.463120 11562 log.cc:1196] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/7442dab63af247d8a842d4e3c2e6b4c8
I20260619 18:41:37.463285 11561 log.cc:1196] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/7442dab63af247d8a842d4e3c2e6b4c8
I20260619 18:41:37.463614 11562 ts_tablet_manager.cc:1939] T 7442dab63af247d8a842d4e3c2e6b4c8 P 3a6f73c544e848be83a8527106fcfed9: Deleting consensus metadata
I20260619 18:41:37.463798 11561 ts_tablet_manager.cc:1939] T 7442dab63af247d8a842d4e3c2e6b4c8 P c0593978efd74090a12433bdf0896f27: Deleting consensus metadata
I20260619 18:41:37.464821 11008 catalog_manager.cc:4841] TS 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885): tablet 7442dab63af247d8a842d4e3c2e6b4c8 (table test-table [id=659e2b392f2d4f058783b05f40eac9d9]) successfully deleted
I20260619 18:41:37.464834 11301 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 (DEFAULT_TABLE table=test-table [id=32a3f204d2574e86b0215694c4b547d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.465317 11301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca9544d7ae54270ba98382c04c19fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.465443 11440 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 (DEFAULT_TABLE table=test-table [id=32a3f204d2574e86b0215694c4b547d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.465849 11440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca9544d7ae54270ba98382c04c19fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.466791 11161 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 (DEFAULT_TABLE table=test-table [id=32a3f204d2574e86b0215694c4b547d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:37.467231 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca9544d7ae54270ba98382c04c19fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:37.472103 11008 catalog_manager.cc:4841] TS c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503): tablet 7442dab63af247d8a842d4e3c2e6b4c8 (table test-table [id=659e2b392f2d4f058783b05f40eac9d9]) successfully deleted
I20260619 18:41:37.472774 11535 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Bootstrap starting.
I20260619 18:41:37.473857 11537 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Bootstrap starting.
I20260619 18:41:37.474821 11535 tablet_bootstrap.cc:654] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.475510 11537 tablet_bootstrap.cc:654] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.476526 11536 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Bootstrap starting.
I20260619 18:41:37.478251 11536 tablet_bootstrap.cc:654] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:37.493242 11535 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: No bootstrap required, opened a new log
I20260619 18:41:37.493463 11535 ts_tablet_manager.cc:1397] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20260619 18:41:37.494315 11535 raft_consensus.cc:348] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.494626 11535 raft_consensus.cc:374] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.494722 11535 raft_consensus.cc:729] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0593978efd74090a12433bdf0896f27, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.506705 11535 consensus_queue.cc:260] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.507437 11535 ts_tablet_manager.cc:1428] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20260619 18:41:37.509174 11537 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: No bootstrap required, opened a new log
I20260619 18:41:37.509387 11537 ts_tablet_manager.cc:1397] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.010s
I20260619 18:41:37.510299 11537 raft_consensus.cc:348] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.510641 11537 raft_consensus.cc:374] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.510741 11537 raft_consensus.cc:729] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.511022 11537 consensus_queue.cc:260] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.511603 11537 ts_tablet_manager.cc:1428] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:37.512508 11536 tablet_bootstrap.cc:492] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: No bootstrap required, opened a new log
I20260619 18:41:37.512701 11536 ts_tablet_manager.cc:1397] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.004s
I20260619 18:41:37.513602 11536 raft_consensus.cc:348] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.513880 11536 raft_consensus.cc:374] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:37.513962 11536 raft_consensus.cc:729] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ef2e84b2204305a21f1b292b0fd930, State: Initialized, Role: FOLLOWER
I20260619 18:41:37.514281 11536 consensus_queue.cc:260] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.514930 11536 ts_tablet_manager.cc:1428] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:37.579677 11543 raft_consensus.cc:482] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:37.579912 11543 raft_consensus.cc:504] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.580708 11543 leader_election.cc:288] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:37.581339 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca9544d7ae54270ba98382c04c19fc3" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef2e84b2204305a21f1b292b0fd930" is_pre_election: true
I20260619 18:41:37.581629 11181 raft_consensus.cc:2454] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 0.
I20260619 18:41:37.581523 11460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca9544d7ae54270ba98382c04c19fc3" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27" is_pre_election: true
I20260619 18:41:37.581815 11460 raft_consensus.cc:2454] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 0.
I20260619 18:41:37.582177 11251 leader_election.cc:302] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:37.582542 11543 raft_consensus.cc:2781] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:37.582693 11543 raft_consensus.cc:482] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:37.582787 11543 raft_consensus.cc:3037] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.586216 11543 raft_consensus.cc:504] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.587061 11543 leader_election.cc:288] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 election: Requested vote from peers 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125), c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503)
I20260619 18:41:37.587772 11460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca9544d7ae54270ba98382c04c19fc3" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0593978efd74090a12433bdf0896f27"
I20260619 18:41:37.587998 11460 raft_consensus.cc:3037] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.588171 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca9544d7ae54270ba98382c04c19fc3" candidate_uuid: "3a6f73c544e848be83a8527106fcfed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ef2e84b2204305a21f1b292b0fd930"
I20260619 18:41:37.588406 11181 raft_consensus.cc:3037] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:37.591172 11460 raft_consensus.cc:2454] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 1.
I20260619 18:41:37.591712 11181 raft_consensus.cc:2454] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a6f73c544e848be83a8527106fcfed9 in term 1.
I20260619 18:41:37.592327 11251 leader_election.cc:302] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ef2e84b2204305a21f1b292b0fd930, 3a6f73c544e848be83a8527106fcfed9; no voters:
I20260619 18:41:37.592773 11543 raft_consensus.cc:2781] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:37.593680 11543 raft_consensus.cc:686] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6f73c544e848be83a8527106fcfed9, State: Running, Role: LEADER
I20260619 18:41:37.594031 11543 consensus_queue.cc:237] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } }
I20260619 18:41:37.600658 11023 catalog_manager.cc:5495] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "3a6f73c544e848be83a8527106fcfed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a6f73c544e848be83a8527106fcfed9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 39885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:37.661731 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.653464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:40813 (local address 127.3.85.129:45125)
0619 18:41:37.653681 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.653694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.654367 (+ 673us) server_negotiation.cc:407] Connection header received
0619 18:41:37.654676 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.654682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.654780 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.654880 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:37.655696 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.656943 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.659413 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.660083 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.660229 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.660879 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.661050 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.661150 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.661326 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.661416 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:41:37.662277 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.652946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45125 (local address 127.3.85.130:40813)
0619 18:41:37.654202 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:37.654224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:37.654479 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:37.654951 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:37.654956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:37.654971 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:41:37.655465 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.655478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.657205 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.657210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.659218 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.659239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.660338 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.660349 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.660552 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.660647 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.661175 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.661188 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:41:37.661277 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:41:37.661456 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":826,"threads_started":1}
I20260619 18:41:37.673969 11460 raft_consensus.cc:1261] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Refusing update from remote peer 3a6f73c544e848be83a8527106fcfed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.673995 11181 raft_consensus.cc:1261] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Refusing update from remote peer 3a6f73c544e848be83a8527106fcfed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:37.674734 11606 consensus_queue.cc:1035] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ef2e84b2204305a21f1b292b0fd930" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.675235 11543 consensus_queue.cc:1035] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0593978efd74090a12433bdf0896f27" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:37.693812 11550 mvcc.cc:205] Tried to move back new op lower bound from 7298639862466777088 to 7298639862148202496. Current Snapshot: MvccSnapshot[applied={T|T < 7298639862466777088}]
I20260619 18:41:37.698235 11551 mvcc.cc:205] Tried to move back new op lower bound from 7298639862466777088 to 7298639862148202496. Current Snapshot: MvccSnapshot[applied={T|T < 7298639862466777088}]
I20260619 18:41:37.722919 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.715969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:46561 (local address 127.3.85.130:39885)
0619 18:41:37.716122 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.716128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.716531 (+ 403us) server_negotiation.cc:407] Connection header received
0619 18:41:37.716722 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.716727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.716814 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.716917 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:37.717481 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.718442 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.720848 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.721468 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.721620 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.722103 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.722206 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.722314 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.722489 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.722566 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:41:37.722977 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.715604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:39885 (local address 127.3.85.129:46561)
0619 18:41:37.716401 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:37.716425 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:37.716581 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:37.716938 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:37.716944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:37.716963 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:41:37.717263 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.717278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.718731 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.718903 (+ 172us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.720674 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:37.720695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.721690 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.721698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:37.721860 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.721902 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:41:37.722316 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:41:37.722329 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:41:37.722442 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:41:37.722607 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":418,"threads_started":1}
I20260619 18:41:37.763573 11024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.763825 11024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:37.767043 11024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d2b56687eca4caab52a417d189d71c5: Sending DeleteTablet for 3 replicas of tablet 6ca9544d7ae54270ba98382c04c19fc3
I20260619 18:41:37.768319 11301 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55270
I20260619 18:41:37.768316 11440 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:55908
I20260619 18:41:37.768636 11561 tablet_replica.cc:330] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: stopping tablet replica
I20260619 18:41:37.768846 11561 raft_consensus.cc:2229] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.769075 11561 raft_consensus.cc:2258] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.769426 11562 tablet_replica.cc:330] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: stopping tablet replica
I20260619 18:41:37.769670 11562 raft_consensus.cc:2229] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:37.770061 11562 raft_consensus.cc:2258] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.770328 11561 ts_tablet_manager.cc:1905] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.771396 11562 ts_tablet_manager.cc:1905] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.777331 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca9544d7ae54270ba98382c04c19fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:37 UTC) from {username='oryx', principal='oryx/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45566
I20260619 18:41:37.777684 11563 tablet_replica.cc:330] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: stopping tablet replica
I20260619 18:41:37.777699 11561 ts_tablet_manager.cc:1918] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.777894 11563 raft_consensus.cc:2229] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:37.777956 11561 log.cc:1196] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/6ca9544d7ae54270ba98382c04c19fc3
I20260619 18:41:37.778108 11563 raft_consensus.cc:2258] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:37.778609 11561 ts_tablet_manager.cc:1939] T 6ca9544d7ae54270ba98382c04c19fc3 P c0593978efd74090a12433bdf0896f27: Deleting consensus metadata
I20260619 18:41:37.779297 11563 ts_tablet_manager.cc:1905] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:37.779541 11562 ts_tablet_manager.cc:1918] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.779711 11562 log.cc:1196] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/6ca9544d7ae54270ba98382c04c19fc3
I20260619 18:41:37.780191 11562 ts_tablet_manager.cc:1939] T 6ca9544d7ae54270ba98382c04c19fc3 P 3a6f73c544e848be83a8527106fcfed9: Deleting consensus metadata
I20260619 18:41:37.780148 11008 catalog_manager.cc:4841] TS c0593978efd74090a12433bdf0896f27 (127.3.85.131:41503): tablet 6ca9544d7ae54270ba98382c04c19fc3 (table test-table [id=32a3f204d2574e86b0215694c4b547d9]) successfully deleted
I20260619 18:41:37.781312 11008 catalog_manager.cc:4841] TS 3a6f73c544e848be83a8527106fcfed9 (127.3.85.130:39885): tablet 6ca9544d7ae54270ba98382c04c19fc3 (table test-table [id=32a3f204d2574e86b0215694c4b547d9]) successfully deleted
I20260619 18:41:37.786305 11563 ts_tablet_manager.cc:1918] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:41:37.786530 11563 log.cc:1196] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/6ca9544d7ae54270ba98382c04c19fc3
I20260619 18:41:37.786947 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11095
I20260619 18:41:37.786990 11563 ts_tablet_manager.cc:1939] T 6ca9544d7ae54270ba98382c04c19fc3 P 36ef2e84b2204305a21f1b292b0fd930: Deleting consensus metadata
I20260619 18:41:37.788246 11008 catalog_manager.cc:4841] TS 36ef2e84b2204305a21f1b292b0fd930 (127.3.85.129:45125): tablet 6ca9544d7ae54270ba98382c04c19fc3 (table test-table [id=32a3f204d2574e86b0215694c4b547d9]) successfully deleted
Jun 19 18:41:37 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.129@KRBTEST.COM
I20260619 18:41:37.804999 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.789063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.3.85.129:45125)
0619 18:41:37.789237 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.789243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.789259 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:41:37.789316 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.789322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.789403 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.789515 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:37.791313 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.792112 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.793570 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:37.793987 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:37.794136 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:37.798766 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:37.798786 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:37.798808 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:37.798850 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:37.801847 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:37.802735 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:37.802744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:37.802746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:37.802885 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:37.803301 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:37.803305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:37.803308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:37.803700 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:37.804066 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:37.804397 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:41:37.804641 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:41:37.805831 11226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:37.958153 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11095
I20260619 18:41:37.997854 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11235
Jun 19 18:41:38 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.130@KRBTEST.COM
I20260619 18:41:38.017234 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:37.998728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.3.85.130:39885)
0619 18:41:37.998921 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:37.998926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:37.999018 (+ 92us) server_negotiation.cc:407] Connection header received
0619 18:41:37.999207 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:37.999211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:37.999279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:37.999425 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.001166 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.001976 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.003985 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.004450 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.004561 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.009878 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.009895 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.009915 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.009971 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.013839 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.014751 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.014760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.014763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.014908 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.015382 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.015386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.015388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.015803 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.016226 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.016581 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.016810 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:41:38.017963 11366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:38.140008 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11235
I20260619 18:41:38.163213 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11375
Jun 19 18:41:38 dist-test-slave-09fp krb5kdc[10979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.85.131@KRBTEST.COM
I20260619 18:41:38.179059 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.163930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.3.85.131:41503)
0619 18:41:38.164359 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.164363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.164381 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:38.164436 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.164441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.164553 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.164670 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.165966 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.166876 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.168117 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.168533 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.168639 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.173128 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.173145 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.173162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.173209 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.176194 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.176940 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.176948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.176950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.177097 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.177461 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.177465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.177467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.177869 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.178097 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.178570 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.178767 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260619 18:41:38.179770 11506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:38.277107 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11375
I20260619 18:41:38.294982 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 10992
I20260619 18:41:38.296164 11064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:38.410919 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 10992
2026-06-19T18:41:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5743 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:38 dist-test-slave-09fp krb5kdc[11636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:38 dist-test-slave-09fp krb5kdc[11636](info): set up 2 sockets
Jun 19 18:41:38 dist-test-slave-09fp krb5kdc[11636](info): commencing operation
krb5kdc: starting...
W20260619 18:41:39.523625 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:39 dist-test-slave-09fp krb5kdc[11636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:39.760080 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35909
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:40.101807 11649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:40.102175 11649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:40.102360 11649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:40.109890 11649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:40.110019 11649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:40.110105 11649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:40.110199 11649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:40.110289 11649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:40.120887 11649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:35909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:40.122328 11649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:40.123770 11649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:40.137143 11657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:40.137115 11655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:40.137677 11654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:40.138077 11649 server_base.cc:1034] running on GCE node
I20260619 18:41:40.138993 11649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:40.140628 11649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:40.141878 11649 hybrid_clock.cc:648] HybridClock initialized: now 1781894500141810 us; error 43 us; skew 500 ppm
I20260619 18:41:40.142299 11649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T18:41:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1797 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:40 dist-test-slave-09fp krb5kdc[11663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:40 dist-test-slave-09fp krb5kdc[11663](info): set up 2 sockets
Jun 19 18:41:40 dist-test-slave-09fp krb5kdc[11663](info): commencing operation
krb5kdc: starting...
W20260619 18:41:41.335505 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:41:41 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:41:41.575870 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37861
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:41.917054 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:41.917447 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:41.917630 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:41.925209 11676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:41.925401 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:41.925518 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:41.925627 11676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:41.925715 11676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:41.936085 11676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:41.937541 11676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:41.938983 11676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:41.951711 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:41.951756 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:41.951782 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:41.952868 11676 server_base.cc:1034] running on GCE node
I20260619 18:41:41.953766 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:41.955688 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:41.956941 11676 hybrid_clock.cc:648] HybridClock initialized: now 1781894501956894 us; error 69 us; skew 500 ppm
I20260619 18:41:41.957332 11676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:41 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:41.965333 11676 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:41.967134 11676 webserver.cc:460] Webserver started at http://127.3.85.190:35507/ using document root <none> and password file <none>
I20260619 18:41:41.967911 11676 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:41.968066 11676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:41.968400 11676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:41.971635 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "84b2f2493dc6468d880aece0168dc356"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-09fp"
server_key: "dfc9f0832e4bdb2b652605a89d138a34"
server_key_iv: "3959901e0702b337013c73d379386d64"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.972499 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "84b2f2493dc6468d880aece0168dc356"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-09fp"
server_key: "dfc9f0832e4bdb2b652605a89d138a34"
server_key_iv: "3959901e0702b337013c73d379386d64"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.978055 11676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:41.981923 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:41.983503 11676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:41.983736 11676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "84b2f2493dc6468d880aece0168dc356"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-09fp"
server_key: "dfc9f0832e4bdb2b652605a89d138a34"
server_key_iv: "3959901e0702b337013c73d379386d64"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.983969 11676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:41.996224 11676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:42.001056 11676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:42.001420 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:42.021417 11676 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37861
I20260619 18:41:42.021446 11742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37861 every 8 connection(s)
I20260619 18:41:42.023084 11676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:42.027150 11743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:42.029309 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11676
I20260619 18:41:42.029516 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:41:42.030330 3414 external_mini_cluster.cc:1428] Setting key f5e3daa90461f1014f0c2f82b739a01e
I20260619 18:41:42.040933 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Bootstrap starting.
I20260619 18:41:42.045255 11743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Neither blocks nor log segments found. Creating new log.
Jun 19 18:41:42 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:42.046584 11743 log.cc:824] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:42.049764 11743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: No bootstrap required, opened a new log
I20260619 18:41:42.055352 11746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:42.032592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.3.85.190:37861)
0619 18:41:42.033495 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:42.033508 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:42.033556 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:41:42.035373 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:42.035398 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:42.035835 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:42.036212 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:42.038140 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.039606 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.041387 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.042121 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.042294 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:42.047347 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:42.047358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:42.047375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:42.047425 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:42.050786 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:42.051588 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:42.051595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:42.051606 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:42.051805 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:42.052222 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:42.052227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:42.052237 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:42.052823 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:42.053179 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:42.053619 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 18:41:42.054094 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":391,"threads_started":1}
I20260619 18:41:42.058740 11743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:42.059206 11743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:42.059315 11743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b2f2493dc6468d880aece0168dc356, State: Initialized, Role: FOLLOWER
I20260619 18:41:42.059965 11743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:42.060233 11743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:42.060375 11743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:42.060559 11743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:42.062943 11743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:42.063534 11743 leader_election.cc:302] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84b2f2493dc6468d880aece0168dc356; no voters:
I20260619 18:41:42.064265 11743 leader_election.cc:288] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:42.064404 11748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:42.065207 11748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 LEADER]: Becoming Leader. State: Replica: 84b2f2493dc6468d880aece0168dc356, State: Running, Role: LEADER
I20260619 18:41:42.065922 11748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:42.066699 11743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:42.069139 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b2f2493dc6468d880aece0168dc356. Latest consensus state: current_term: 1 leader_uuid: "84b2f2493dc6468d880aece0168dc356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } } }
I20260619 18:41:42.069666 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:42.069689 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84b2f2493dc6468d880aece0168dc356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } } }
I20260619 18:41:42.070024 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:42.072338 11753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:42.077966 11753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:42.088374 11753 catalog_manager.cc:1347] Generated new cluster ID: 36d4891c63cf4946b046fcb56e7f7dc1
I20260619 18:41:42.088495 11753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:42.109711 11753 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:42.111179 11753 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:42.124176 11753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Generated new TSK 0
I20260619 18:41:42.125123 11753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:41:42.190541 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:42.552836 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:42.553153 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:42.553252 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:42.561017 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:42.561157 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:42.561273 11769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:42.572033 11769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:42.573508 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:42.574910 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:42.589578 11774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:42.589668 11775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:42.589609 11777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:42.590646 11769 server_base.cc:1034] running on GCE node
I20260619 18:41:42.591372 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:42.592482 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:42.593708 11769 hybrid_clock.cc:648] HybridClock initialized: now 1781894502593665 us; error 55 us; skew 500 ppm
I20260619 18:41:42.594067 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:42 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:42.597568 11769 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:41:42.599296 11769 webserver.cc:460] Webserver started at http://127.3.85.129:36815/ using document root <none> and password file <none>
I20260619 18:41:42.600090 11769 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:42.600260 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:42.600587 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:42.603802 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "c796fdd6232d4d72a6ff72af4578f157"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-09fp"
server_key: "6afbef2c53a3db5c9698d77e8e47b846"
server_key_iv: "a4d290e3a8179691b939e2d3df533c86"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.604718 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "c796fdd6232d4d72a6ff72af4578f157"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-09fp"
server_key: "6afbef2c53a3db5c9698d77e8e47b846"
server_key_iv: "a4d290e3a8179691b939e2d3df533c86"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.610663 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:41:42.614612 11784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:42.616194 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:42.616434 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "c796fdd6232d4d72a6ff72af4578f157"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-09fp"
server_key: "6afbef2c53a3db5c9698d77e8e47b846"
server_key_iv: "a4d290e3a8179691b939e2d3df533c86"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.616683 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:42.636178 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:42.640902 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:42.641270 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:42.642355 11769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:42.643862 11769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:42.643985 11769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:42.644153 11769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:42.644234 11769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:42.680258 11769 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:35113
I20260619 18:41:42.680287 11896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:35113 every 8 connection(s)
I20260619 18:41:42.681665 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:42.688938 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11769
I20260619 18:41:42.689183 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:42.689898 3414 external_mini_cluster.cc:1428] Setting key 40d1c5067989f176bcb2fd54a46d926c
Jun 19 18:41:42 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:42.705325 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:42.684490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:36811 (local address 127.3.85.190:37861)
0619 18:41:42.684951 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:42.684956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:42.686050 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 18:41:42.688069 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:42.688075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:42.688158 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:42.688380 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:42.690198 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.691142 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.693206 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.694065 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.694252 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:42.698079 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:42.698089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:42.698097 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:42.698153 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:42.701255 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:42.702600 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:42.702605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:42.702607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:42.702749 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:42.703339 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:42.703342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:42.703344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:42.703619 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:42.703750 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:42.704832 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0619 18:41:42.705055 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260619 18:41:42.707099 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:42.685112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:36811)
0619 18:41:42.685853 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:42.685899 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:42.687695 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:42.688609 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:42.688618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:42.689311 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:42.689979 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:42.690008 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.691327 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.691333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:42.693015 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:42.693032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.694332 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.694338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:42.694654 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:42.695401 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:42.695431 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:42.697767 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:42.701480 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:42.701496 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:42.701521 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:42.702345 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:42.702893 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:42.702902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:42.702909 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:42.703203 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:42.703765 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:42.703778 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:42.704352 (+ 574us) client_negotiation.cc:769] Sending connection context
0619 18:41:42.704715 (+ 363us) client_negotiation.cc:240] Negotiation successful
0619 18:41:42.705388 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":290,"threads_started":1}
I20260619 18:41:42.712098 11897 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37861
I20260619 18:41:42.712543 11897 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:42.713614 11897 heartbeater.cc:511] Master 127.3.85.190:37861 requested a full tablet report, sending...
I20260619 18:41:42.717149 11708 ts_manager.cc:194] Registered new tserver with Master: c796fdd6232d4d72a6ff72af4578f157 (127.3.85.129:35113)
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:41:42.719691 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:36811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:41:42.750231 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:43.106693 11903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:43.107007 11903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:43.107178 11903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:43.115417 11903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:43.115547 11903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:43.115761 11903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:43.127115 11903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:43.128500 11903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:43.129940 11903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:43.144197 11911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:43.144304 11909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:43.144287 11908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:43.148284 11903 server_base.cc:1034] running on GCE node
I20260619 18:41:43.149025 11903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:43.150100 11903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:43.151324 11903 hybrid_clock.cc:648] HybridClock initialized: now 1781894503151276 us; error 43 us; skew 500 ppm
I20260619 18:41:43.151729 11903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:43 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:43.155195 11903 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:41:43.156960 11903 webserver.cc:460] Webserver started at http://127.3.85.130:43911/ using document root <none> and password file <none>
I20260619 18:41:43.157812 11903 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:43.157984 11903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:43.158327 11903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:43.161554 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "c5d0b5aed94240059d3fc0fc5e7ec00a"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "e2ff42e6197fd5d3fd79b8562082fb6e"
server_key_iv: "9feecdb3896317d760a33a630e3b11ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.162618 11903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "c5d0b5aed94240059d3fc0fc5e7ec00a"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "e2ff42e6197fd5d3fd79b8562082fb6e"
server_key_iv: "9feecdb3896317d760a33a630e3b11ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.168658 11903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:41:43.172947 11918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:43.174769 11903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:43.175012 11903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "c5d0b5aed94240059d3fc0fc5e7ec00a"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "e2ff42e6197fd5d3fd79b8562082fb6e"
server_key_iv: "9feecdb3896317d760a33a630e3b11ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.175261 11903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:43.192163 11903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:43.196939 11903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:43.197279 11903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:43.198350 11903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:43.199851 11903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:43.199963 11903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:43.200126 11903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:43.200198 11903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:43.237345 11903 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:36665
I20260619 18:41:43.237396 12030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:36665 every 8 connection(s)
I20260619 18:41:43.238843 11903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:43.243927 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 11903
I20260619 18:41:43.244120 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:43.244680 3414 external_mini_cluster.cc:1428] Setting key c8d568cc3355fff9d753927c0aa8d144
Jun 19 18:41:43 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:43.263520 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:43.241592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51739 (local address 127.3.85.190:37861)
0619 18:41:43.242051 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:43.242056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:43.243159 (+ 1103us) server_negotiation.cc:407] Connection header received
0619 18:41:43.245787 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:43.245792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:43.245897 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:43.246027 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:43.248289 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.249095 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.251949 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.252413 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.252526 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:43.257173 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:43.257178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:43.257181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:43.257216 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:43.259934 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.260818 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.260823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.260826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.260966 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.261519 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.261524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.261526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.261815 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:43.262006 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:43.262855 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 18:41:43.263125 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260619 18:41:43.264628 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:43.242186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:51739)
0619 18:41:43.242978 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:43.243026 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:43.245466 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:43.246300 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:43.246310 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:43.247085 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:43.248030 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:43.248064 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.249318 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.249326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:43.251733 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:43.251757 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.252686 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.252697 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:43.253058 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:43.253943 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:43.253978 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:43.256870 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:43.260125 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:43.260135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:43.260165 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:43.260653 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:43.261095 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:43.261100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:43.261103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:43.261368 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:43.261968 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:43.261978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:43.262375 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 18:41:43.262742 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 18:41:43.263350 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":339,"threads_started":1}
I20260619 18:41:43.266650 12031 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37861
I20260619 18:41:43.267031 12031 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:43.267825 12031 heartbeater.cc:511] Master 127.3.85.190:37861 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:41:43.269608 11708 ts_manager.cc:194] Registered new tserver with Master: c5d0b5aed94240059d3fc0fc5e7ec00a (127.3.85.130:36665)
I20260619 18:41:43.270581 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:51739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:41:43.299726 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:43.644752 12037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:43.645074 12037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:43.645171 12037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:43.652756 12037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:43.652909 12037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:43.653028 12037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:43.664791 12037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:43.666139 12037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:43.667685 12037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:43.683445 12043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:43.683422 12042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:43.684415 12037 server_base.cc:1034] running on GCE node
W20260619 18:41:43.687053 12045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:43.687963 12037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:43.689112 12037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:43.690378 12037 hybrid_clock.cc:648] HybridClock initialized: now 1781894503690300 us; error 67 us; skew 500 ppm
I20260619 18:41:43.690855 12037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:43 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:43.694427 12037 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:41:43.696136 12037 webserver.cc:460] Webserver started at http://127.3.85.131:35739/ using document root <none> and password file <none>
I20260619 18:41:43.696916 12037 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:43.697028 12037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:43.697306 12037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:43.700557 12037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "d2e092f9c186422a951cd019657b6129"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "a7b688db6fd5b8989bfe3d6812110540"
server_key_iv: "fcfed70fe680f1bfea51dd5d6b6c04c9"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.701495 12037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e092f9c186422a951cd019657b6129"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "a7b688db6fd5b8989bfe3d6812110540"
server_key_iv: "fcfed70fe680f1bfea51dd5d6b6c04c9"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.707546 12037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:41:43.711512 12052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:43.713100 12037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:43.713271 12037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "d2e092f9c186422a951cd019657b6129"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "a7b688db6fd5b8989bfe3d6812110540"
server_key_iv: "fcfed70fe680f1bfea51dd5d6b6c04c9"
server_key_version: "encryptionkey@0"
I20260619 18:41:43.713464 12037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:43.723621 11897 heartbeater.cc:503] Master 127.3.85.190:37861 was elected leader, sending a full tablet report...
I20260619 18:41:43.734169 12037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:43.739215 12037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:43.739537 12037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:43.740628 12037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:43.742206 12037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:43.742326 12037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:43.742430 12037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:43.742539 12037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 18:41:43.777760 12037 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:34977
I20260619 18:41:43.777809 12164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:34977 every 8 connection(s)
I20260619 18:41:43.779305 12037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:43.783406 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12037
I20260619 18:41:43.783592 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:43.784183 3414 external_mini_cluster.cc:1428] Setting key 8d9ca2f145ff92b2b1d41742383b2f6a
Jun 19 18:41:43 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:41:43.803083 12167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:43.782222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:42739 (local address 127.3.85.190:37861)
0619 18:41:43.782714 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:43.782718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:43.784016 (+ 1298us) server_negotiation.cc:407] Connection header received
0619 18:41:43.786538 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:43.786543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:43.786691 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:43.786834 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:43.788923 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.789707 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.791511 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.791931 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.792089 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:43.796053 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:43.796058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:43.796062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:43.796106 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:43.799636 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.800750 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.800759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.800765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.800935 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.801350 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.801353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.801355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.801636 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:43.801758 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:43.802589 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 18:41:43.802784 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":286,"threads_started":1}
I20260619 18:41:43.804342 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:43.782816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:42739)
0619 18:41:43.783780 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:43.783847 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:43.786190 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:43.787214 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:43.787223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:43.788006 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:43.788724 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:43.788748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.789892 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.789896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:43.791315 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:43.791331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.792207 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.792215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:43.792482 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:43.793246 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:43.793278 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:43.795749 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:43.799843 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:43.799852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:43.799896 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:43.800591 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:43.801054 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:43.801058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:43.801060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:43.801248 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:43.801746 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:43.801752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:43.802120 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 18:41:43.802360 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 18:41:43.802867 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":348,"threads_started":1}
I20260619 18:41:43.805853 12165 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37861
I20260619 18:41:43.806216 12165 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:43.807072 12165 heartbeater.cc:511] Master 127.3.85.190:37861 requested a full tablet report, sending...
I20260619 18:41:43.808686 11708 ts_manager.cc:194] Registered new tserver with Master: d2e092f9c186422a951cd019657b6129 (127.3.85.131:34977)
I20260619 18:41:43.809551 11708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:42739
I20260619 18:41:43.813354 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
Entry for principal oryx/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab.
WARNING: no policy specified for oryx/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
Entry for principal oryx/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab.
WARNING: no policy specified for oryx/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
Entry for principal oryx/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab.
I20260619 18:41:43.963579 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11769
Jun 19 18:41:43 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:41:43.981421 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:43.964910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.3.85.129:35113)
0619 18:41:43.965563 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:43.965570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:43.965600 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:41:43.965695 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:43.965702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:43.965921 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:43.966064 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:43.967688 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.968845 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.970102 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:43.970756 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:43.970865 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:43.975129 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:43.975143 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:43.975160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:43.975200 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:43.977992 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.979130 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.979141 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.979150 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.979323 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:43.979809 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:43.979813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:43.979815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:43.980308 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:43.980568 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:43.980911 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:41:43.981120 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":326,"threads_started":1}
I20260619 18:41:43.982784 11892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:44.074791 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11769
I20260619 18:41:44.093204 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11903
Jun 19 18:41:44 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:41:44.111933 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:44.094317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.3.85.130:36665)
0619 18:41:44.094897 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:44.094904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:44.094934 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:41:44.095004 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:44.095012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:44.095243 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:44.095410 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:44.097143 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:44.098192 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:44.099950 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:44.100546 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:44.100674 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:44.105145 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:44.105161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:44.105179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:44.105221 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:44.108652 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:44.109546 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:44.109556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:44.109558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:44.109714 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:44.110160 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:44.110164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:44.110167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:44.110633 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:44.110924 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:44.111380 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 18:41:44.111621 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":303,"threads_started":1}
I20260619 18:41:44.113265 12026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:44.201654 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11903
I20260619 18:41:44.218411 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12037
Jun 19 18:41:44 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:41:44.236167 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:44.219448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.3.85.131:34977)
0619 18:41:44.219920 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:44.219927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:44.219966 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:41:44.220028 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:44.220036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:44.220248 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:44.220440 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:44.221820 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:44.222881 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:44.224206 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:44.224785 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:44.224959 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:44.229926 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:44.229950 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:44.229967 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:44.230008 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:44.233022 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:44.233873 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:44.233882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:44.233884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:44.234030 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:44.234496 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:44.234501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:44.234503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:44.234876 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:44.235195 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:44.235511 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 18:41:44.235781 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":206,"threads_started":1}
I20260619 18:41:44.237531 12160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:44.321435 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12037
I20260619 18:41:44.337612 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 11676
I20260619 18:41:44.338869 11738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:41:44.442729 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 11676
I20260619 18:41:44.464221 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37861
--webserver_interface=127.3.85.190
--webserver_port=35507
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:44.810027 12183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:44.810367 12183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:44.810563 12183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:44.818240 12183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:44.818372 12183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:44.818460 12183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:44.818600 12183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:44.818684 12183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:44.829336 12183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=35507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:44.830794 12183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:44.832211 12183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:44.845454 12189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:44.845850 12188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:44.845956 12191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:44.846493 12183 server_base.cc:1034] running on GCE node
I20260619 18:41:44.847494 12183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:44.849537 12183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:44.850781 12183 hybrid_clock.cc:648] HybridClock initialized: now 1781894504850700 us; error 55 us; skew 500 ppm
I20260619 18:41:44.851245 12183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:44 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:44.854779 12183 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:41:44.856487 12183 webserver.cc:460] Webserver started at http://127.3.85.190:35507/ using document root <none> and password file <none>
I20260619 18:41:44.857285 12183 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:44.857460 12183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:44.863723 12183 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:44.867301 12198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:44.869058 12183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:44.869335 12183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "84b2f2493dc6468d880aece0168dc356"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-09fp"
server_key: "dfc9f0832e4bdb2b652605a89d138a34"
server_key_iv: "3959901e0702b337013c73d379386d64"
server_key_version: "encryptionkey@0"
I20260619 18:41:44.870452 12183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:44.892940 12183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:44.897859 12183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:44.898196 12183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:44.919063 12183 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37861
I20260619 18:41:44.919086 12249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37861 every 8 connection(s)
I20260619 18:41:44.920706 12183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:44.927667 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12183
I20260619 18:41:44.928689 12250 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:41:44.928705 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:35113
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:37861
--webserver_port=36815
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:44.934557 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Bootstrap starting.
I20260619 18:41:44.955725 12250 log.cc:824] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:44.965595 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:41:44.966267 12250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Bootstrap complete.
I20260619 18:41:44.976758 12250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:44.977656 12250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b2f2493dc6468d880aece0168dc356, State: Initialized, Role: FOLLOWER
I20260619 18:41:44.978395 12250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:44.978717 12250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:44.978871 12250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:44.979059 12250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:41:44.983842 12250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:44.984472 12250 leader_election.cc:302] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84b2f2493dc6468d880aece0168dc356; no voters:
I20260619 18:41:44.985153 12250 leader_election.cc:288] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:41:44.985306 12254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:41:44.986742 12254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [term 2 LEADER]: Becoming Leader. State: Replica: 84b2f2493dc6468d880aece0168dc356, State: Running, Role: LEADER
I20260619 18:41:44.987416 12254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } }
I20260619 18:41:44.987886 12250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:44.990823 12256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b2f2493dc6468d880aece0168dc356. Latest consensus state: current_term: 2 leader_uuid: "84b2f2493dc6468d880aece0168dc356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } } }
I20260619 18:41:44.991218 12256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:44.990355 12255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "84b2f2493dc6468d880aece0168dc356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b2f2493dc6468d880aece0168dc356" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37861 } } }
I20260619 18:41:44.991910 12255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:44.996165 12261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:45.003132 12261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:45.004771 12261 catalog_manager.cc:1259] Loaded cluster ID: 36d4891c63cf4946b046fcb56e7f7dc1
I20260619 18:41:45.004904 12261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:45.009395 12261 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:45.011209 12261 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 84b2f2493dc6468d880aece0168dc356: Loaded TSK: 0
I20260619 18:41:45.012890 12261 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:41:45.312630 12252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:45.313045 12252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:45.313231 12252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:45.321170 12252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:45.321386 12252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:45.321596 12252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:41:45.332422 12252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.129.keytab
--principal=oryx/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:35113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=36815
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:45.333997 12252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:45.335563 12252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:45.353612 12276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.353612 12277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.354513 12279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:45.355108 12252 server_base.cc:1034] running on GCE node
I20260619 18:41:45.355935 12252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:45.357249 12252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:45.358536 12252 hybrid_clock.cc:648] HybridClock initialized: now 1781894505358464 us; error 69 us; skew 500 ppm
I20260619 18:41:45.358937 12252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894505, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:45.363351 12252 init.cc:377] Logged in from keytab as oryx/127.3.85.129@KRBTEST.COM (short username oryx)
I20260619 18:41:45.365614 12252 webserver.cc:460] Webserver started at http://127.3.85.129:36815/ using document root <none> and password file <none>
I20260619 18:41:45.366667 12252 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:45.366854 12252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:45.373409 12252 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:45.377069 12286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.378837 12252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:41:45.379045 12252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "c796fdd6232d4d72a6ff72af4578f157"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-09fp"
server_key: "6afbef2c53a3db5c9698d77e8e47b846"
server_key_iv: "a4d290e3a8179691b939e2d3df533c86"
server_key_version: "encryptionkey@0"
I20260619 18:41:45.380119 12252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:45.398130 12252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:45.403308 12252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:45.403623 12252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:45.404716 12252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:45.406234 12252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:45.406333 12252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.406513 12252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:45.406594 12252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.442819 12252 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:35113
I20260619 18:41:45.442951 12398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:35113 every 8 connection(s)
I20260619 18:41:45.444448 12252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:45.446624 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12252
I20260619 18:41:45.447347 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:36665
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:37861
--webserver_port=43911
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:45.465858 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.448106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:51879)
0619 18:41:45.449697 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.449752 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.451566 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.453390 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.453404 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.454054 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.454729 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.454751 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.456546 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.456557 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.457933 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.457948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.459176 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.459184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.459466 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.460219 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.460244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.464362 (+ 4118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1311,"thread_start_us":1123,"threads_started":1}
W20260619 18:41:45.465888 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.448058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:51879 (local address 127.3.85.190:37861)
0619 18:41:45.449052 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.449076 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.449926 (+ 850us) server_negotiation.cc:407] Connection header received
0619 18:41:45.452235 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.452258 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.452747 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.453169 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.454952 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.456325 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.458111 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.458898 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.459124 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.464785 (+ 5661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:51879: BlockingRecv error: recv got EOF from 127.3.85.129:51879 (error 108)
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":334,"threads_started":1}
W20260619 18:41:45.467121 12399 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:37861 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.85.190:37861: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:45.478379 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.467895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:55949)
0619 18:41:45.468232 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.468255 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.468417 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.468768 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.468776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.469289 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.469539 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.469555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.470760 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.470768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.472319 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.472337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.473227 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.473234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.473460 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.474221 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.474252 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.478064 (+ 3812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186}
W20260619 18:41:45.478523 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.468015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:55949 (local address 127.3.85.190:37861)
0619 18:41:45.468182 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.468186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.468345 (+ 159us) server_negotiation.cc:407] Connection header received
0619 18:41:45.468542 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.468546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.468606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.468698 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.469740 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.470598 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.472606 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.473056 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.473173 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.478284 (+ 5111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:55949: BlockingRecv error: recv got EOF from 127.3.85.129:55949 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:45.488473 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.479132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:58983)
0619 18:41:45.479357 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.479383 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.479481 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.479999 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.480003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.480296 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.480469 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.480477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.481956 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.481960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.483088 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.483102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.484095 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.484103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.484336 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.484982 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.485001 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.488151 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:41:45.488659 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.479263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:58983 (local address 127.3.85.190:37861)
0619 18:41:45.479474 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.479480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.479502 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:45.479566 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.479571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.479663 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.479768 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.480621 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.481775 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.483233 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.483866 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.484006 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.488385 (+ 4379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:58983: BlockingRecv error: recv got EOF from 127.3.85.129:58983 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:41:45.488811 12399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:41:45.808174 12403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:45.808492 12403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:45.808658 12403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:45.816386 12403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:45.816524 12403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:45.816716 12403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:41:45.827095 12403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.130.keytab
--principal=oryx/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:36665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=43911
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:45.828577 12403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:45.829991 12403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:45.844762 12409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.844791 12408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.844837 12411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:45.845732 12403 server_base.cc:1034] running on GCE node
I20260619 18:41:45.846515 12403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:45.847937 12403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:45.849287 12403 hybrid_clock.cc:648] HybridClock initialized: now 1781894505849225 us; error 65 us; skew 500 ppm
I20260619 18:41:45.849686 12403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894505, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:45.854589 12403 init.cc:377] Logged in from keytab as oryx/127.3.85.130@KRBTEST.COM (short username oryx)
I20260619 18:41:45.856555 12403 webserver.cc:460] Webserver started at http://127.3.85.130:43911/ using document root <none> and password file <none>
I20260619 18:41:45.857359 12403 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:45.857528 12403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:45.863945 12403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:41:45.867337 12418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.869305 12403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:45.869517 12403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "c5d0b5aed94240059d3fc0fc5e7ec00a"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "e2ff42e6197fd5d3fd79b8562082fb6e"
server_key_iv: "9feecdb3896317d760a33a630e3b11ff"
server_key_version: "encryptionkey@0"
I20260619 18:41:45.870826 12403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:45.896873 12403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:45.901813 12403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:45.902081 12403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:45.903450 12403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:45.904943 12403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:45.905059 12403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.905138 12403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:45.905192 12403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.944185 12403 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:36665
I20260619 18:41:45.944393 12530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:36665 every 8 connection(s)
I20260619 18:41:45.945729 12403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:45.954167 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12403
I20260619 18:41:45.954941 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:34977
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:37861
--webserver_port=35739
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:41:45.963769 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.948735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:50385 (local address 127.3.85.190:37861)
0619 18:41:45.948944 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.948951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.950295 (+ 1344us) server_negotiation.cc:407] Connection header received
0619 18:41:45.952194 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.952199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.952269 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.952374 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.953917 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.955063 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.957009 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.957762 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.957918 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.963525 (+ 5607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:50385: BlockingRecv error: recv got EOF from 127.3.85.130:50385 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:41:45.964605 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.949298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:50385)
0619 18:41:45.950092 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.950137 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.951909 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.952601 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.952610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.953138 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.953726 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.953743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.955326 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.955332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.956823 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.956838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.958027 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.958032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.958255 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.959126 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.959150 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.963345 (+ 4195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":332,"threads_started":1}
W20260619 18:41:45.965561 12531 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:37861 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.85.190:37861: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:45.974241 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.966230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:41035)
0619 18:41:45.966495 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.966516 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.966635 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.966981 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.966986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.967347 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.967484 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.967493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.968511 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.968517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.969421 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.969432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.970182 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.970188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.970371 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.971040 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.971057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.973935 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 18:41:45.974414 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.966395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:41035 (local address 127.3.85.190:37861)
0619 18:41:45.966599 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.966604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.966623 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:45.966786 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.966790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.966852 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.966955 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.967631 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.968369 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.969574 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.970011 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.970153 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.974162 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:41035: BlockingRecv error: recv got EOF from 127.3.85.130:41035 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:45 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:45.984553 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.975462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:37225)
0619 18:41:45.975623 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:45.975642 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:45.975781 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:45.976166 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:45.976171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:45.976529 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:45.976732 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.976744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.977984 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.977990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.979111 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:45.979128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.979912 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.979919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:45.980129 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.980831 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:45.980851 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:45.984236 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 18:41:45.984921 12531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:41:45.985317 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:45.975663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:37225 (local address 127.3.85.190:37861)
0619 18:41:45.975874 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.975881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.975899 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:45.975960 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.975963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.976036 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.976138 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.976894 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.977790 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.979307 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.979736 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.979895 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.984577 (+ 4682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:37225: BlockingRecv error: recv got EOF from 127.3.85.130:37225 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 18:41:46.316859 12534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:46.317178 12534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:46.317358 12534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:46.324891 12534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:46.325024 12534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:46.325229 12534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:41:46.335458 12534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/krb5kdc/oryx_127.3.85.131.keytab
--principal=oryx/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=35739
--tserver_master_addrs=127.3.85.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:41:46.336920 12534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:46.338268 12534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:46.352408 12539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:46.352496 12542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:46.352463 12540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:46.353857 12534 server_base.cc:1034] running on GCE node
I20260619 18:41:46.354621 12534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:46.355650 12534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:46.356874 12534 hybrid_clock.cc:648] HybridClock initialized: now 1781894506356828 us; error 48 us; skew 500 ppm
I20260619 18:41:46.357210 12534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894506, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:46.360558 12534 init.cc:377] Logged in from keytab as oryx/127.3.85.131@KRBTEST.COM (short username oryx)
I20260619 18:41:46.362186 12534 webserver.cc:460] Webserver started at http://127.3.85.131:35739/ using document root <none> and password file <none>
I20260619 18:41:46.363018 12534 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:46.363175 12534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:46.369030 12534 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:46.372066 12549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:46.373529 12534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:41:46.373740 12534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "d2e092f9c186422a951cd019657b6129"
format_stamp: "Formatted at 2026-06-19 18:41:43 on dist-test-slave-09fp"
server_key: "a7b688db6fd5b8989bfe3d6812110540"
server_key_iv: "fcfed70fe680f1bfea51dd5d6b6c04c9"
server_key_version: "encryptionkey@0"
I20260619 18:41:46.374897 12534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:46.390718 12534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:46.395644 12534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:46.395960 12534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:46.397051 12534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:46.398576 12534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:46.398690 12534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:46.398840 12534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:46.398934 12534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:46.433017 12534 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:34977
I20260619 18:41:46.433143 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:34977 every 8 connection(s)
I20260619 18:41:46.434654 12534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:46.440694 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12534
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:41:46.454180 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:46.437383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:48195 (local address 127.3.85.190:37861)
0619 18:41:46.437586 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.437591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.439047 (+ 1456us) server_negotiation.cc:407] Connection header received
0619 18:41:46.441001 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.441006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.441101 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.441220 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.443508 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.444689 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.447203 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.447948 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.448057 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.453905 (+ 5848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:48195: BlockingRecv error: recv got EOF from 127.3.85.131:48195 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:41:46.455073 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.438020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:48195)
0619 18:41:46.438848 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:46.438889 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:46.440656 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:46.441520 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:46.441533 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:46.442436 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:46.443266 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.443298 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.444942 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.444948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.446996 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.447014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.448509 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.448516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.448826 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.449687 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:46.449732 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:46.453720 (+ 3988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":293,"threads_started":1}
W20260619 18:41:46.455766 12662 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:37861 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.85.190:37861: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:46.465445 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.456410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:37817)
0619 18:41:46.456616 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:46.456635 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:46.456747 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:46.457075 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:46.457081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:46.457553 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:46.457779 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.457795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.459011 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.459015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.459914 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.459926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.461031 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.461039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.461223 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.461769 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:46.461787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:46.465136 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:41:46.465621 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:46.456511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:37817 (local address 127.3.85.190:37861)
0619 18:41:46.456669 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.456673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.456689 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:41:46.456870 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.456874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.456930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.457008 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.457951 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.458835 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.460067 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.460848 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.460956 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.465376 (+ 4420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:37817: BlockingRecv error: recv got EOF from 127.3.85.131:37817 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:41:46.466404 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.441846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.3.85.190:37861)
0619 18:41:46.449734 (+ 7888us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.449741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.450232 (+ 491us) server_negotiation.cc:407] Connection header received
0619 18:41:46.450337 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.450343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.450423 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.450569 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.452284 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.453212 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.454720 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.455178 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.455319 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.458838 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:46.458850 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:46.458878 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:46.458933 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:46.462509 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:46.463356 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:46.463367 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:46.463378 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:46.463643 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:46.464082 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:46.464087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:46.464091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:46.464731 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:46.464987 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:46.465586 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 18:41:46.466001 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7752,"thread_start_us":7622,"threads_started":1}
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:46.474924 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.466287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:56963)
0619 18:41:46.466500 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:46.466521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:46.466642 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:46.467167 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:46.467170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:46.467457 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:46.467609 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.467618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.468745 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.468751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.469895 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.469909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.470797 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.470802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.470981 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.471556 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:46.471574 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:46.474581 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:41:46.475209 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:46.466640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:56963 (local address 127.3.85.190:37861)
0619 18:41:46.466874 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.466880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.466895 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:41:46.466948 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.466955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.467028 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.467132 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.467818 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.468581 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.470055 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.470623 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.470755 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.474958 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:56963: BlockingRecv error: recv got EOF from 127.3.85.131:56963 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260619 18:41:46.475255 12662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:46.498880 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.489421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:60881)
0619 18:41:46.490029 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:46.490046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:46.490223 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:46.490647 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:46.490652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:46.491145 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:46.491333 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.491342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.492381 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.492387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.493356 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.493367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.494259 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.494266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.494437 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.495160 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:46.495181 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:46.498615 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":347,"threads_started":1}
W20260619 18:41:46.499122 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:46.489642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:60881 (local address 127.3.85.190:37861)
0619 18:41:46.489894 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.489900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.490146 (+ 246us) server_negotiation.cc:407] Connection header received
0619 18:41:46.490370 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.490375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.490436 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.490609 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.491493 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.492225 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.493507 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.494052 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.494212 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.498863 (+ 4651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:60881: BlockingRecv error: recv got EOF from 127.3.85.129:60881 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:46 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:46.998414 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.985555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:52713)
0619 18:41:46.986143 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:46.986164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:46.986287 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:46.986697 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:46.986702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:46.987294 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:46.987552 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.987565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.988659 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.988665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.989733 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:46.989746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.990564 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.990571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:46.990740 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.994877 (+ 4137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:46.994900 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:46.998177 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
W20260619 18:41:46.998663 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:46.985643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:52713 (local address 127.3.85.190:37861)
0619 18:41:46.985815 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.985819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.986247 (+ 428us) server_negotiation.cc:407] Connection header received
0619 18:41:46.986401 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.986408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.986527 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.986657 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.987705 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.988504 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.989897 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.990319 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.990458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.998377 (+ 7919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:52713: BlockingRecv error: recv got EOF from 127.3.85.130:52713 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:41:47 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:47 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:47.485347 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:47.476002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:36465)
0619 18:41:47.476545 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:47.476562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:47.476715 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:47.477111 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:47.477116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:47.477587 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:47.477779 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:47.477789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.479308 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.479312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:47.480451 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:47.480467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.481211 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.481216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:47.481355 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:47.481970 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:47.481987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:47.485097 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":293,"threads_started":1}
W20260619 18:41:47.485744 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:47.476060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:36465 (local address 127.3.85.190:37861)
0619 18:41:47.476273 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:47.476279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:47.476649 (+ 370us) server_negotiation.cc:407] Connection header received
0619 18:41:47.476821 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:47.476825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:47.476886 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:47.477009 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:47.477937 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.478980 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.480625 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.481033 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.481144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:47.485492 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:36465: BlockingRecv error: recv got EOF from 127.3.85.131:36465 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 18:41:47 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:47 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:47.510803 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:47.499936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:48509)
0619 18:41:47.500563 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:47.500587 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:47.500742 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:47.501162 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:47.501166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:47.501648 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:47.501815 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:47.501826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.502970 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.502974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:47.503886 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:47.503899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.505235 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.505243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:47.505451 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:47.506407 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:47.506437 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:47.510539 (+ 4102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":344,"threads_started":1}
W20260619 18:41:47.511168 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:47.500035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:48509 (local address 127.3.85.190:37861)
0619 18:41:47.500276 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:47.500285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:47.500728 (+ 443us) server_negotiation.cc:407] Connection header received
0619 18:41:47.500882 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:47.500888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:47.500985 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:47.501098 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:47.501961 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.502828 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.504054 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:47.504962 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:47.505213 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:47.510820 (+ 5607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:48509: BlockingRecv error: recv got EOF from 127.3.85.129:48509 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:48.008366 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:47.999416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:34183)
0619 18:41:47.999950 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:47.999970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:48.000135 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:48.000504 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:48.000510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:48.000954 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:48.001096 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.001106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.002190 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.002197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.003190 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.003203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.003963 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.003969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.004133 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.004812 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:48.004830 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:48.008005 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":311,"threads_started":1}
W20260619 18:41:48.008484 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:47.999538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:34183 (local address 127.3.85.190:37861)
0619 18:41:47.999766 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:47.999775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:48.000058 (+ 283us) server_negotiation.cc:407] Connection header received
0619 18:41:48.000267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:48.000273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:48.000356 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:48.000463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:48.001273 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.002040 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.003349 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.003795 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.003939 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.008256 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:34183: BlockingRecv error: recv got EOF from 127.3.85.130:34183 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:48.495544 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:48.486536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:42547)
0619 18:41:48.487125 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:48.487142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:48.487296 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:48.487698 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:48.487703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:48.488096 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:48.488235 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.488243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.489483 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.489487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.490524 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.490549 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.491282 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.491287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.491411 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.492100 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:48.492119 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:48.495170 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
W20260619 18:41:48.495723 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:48.486650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:42547 (local address 127.3.85.190:37861)
0619 18:41:48.486849 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:48.486857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:48.487222 (+ 365us) server_negotiation.cc:407] Connection header received
0619 18:41:48.487426 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:48.487431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:48.487558 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:48.487685 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:48.488388 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.489312 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.490680 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.491119 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.491250 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.495479 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:42547: BlockingRecv error: recv got EOF from 127.3.85.131:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:48 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:48.520393 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:48.511817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:36257)
0619 18:41:48.512346 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:48.512367 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:48.512522 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:48.512852 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:48.512857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:48.513240 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:48.513401 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.513410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.514412 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.514419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.515375 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:48.515387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.516087 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.516093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:48.516277 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.516900 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:48.516919 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:48.520094 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
W20260619 18:41:48.520821 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:48.511893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:36257 (local address 127.3.85.190:37861)
0619 18:41:48.512085 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:48.512093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:48.512455 (+ 362us) server_negotiation.cc:407] Connection header received
0619 18:41:48.512645 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:48.512651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:48.512724 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:48.512828 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:48.513549 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.514262 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.515538 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:48.515933 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:48.516070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:48.520361 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:36257: BlockingRecv error: recv got EOF from 127.3.85.129:36257 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:49.018883 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:49.009496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:49897)
0619 18:41:49.009999 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:49.010020 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:49.010153 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:49.010537 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:49.010540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:49.010892 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:49.011087 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.011096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.012393 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.012399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.013362 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.013376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.014118 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.014123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.014256 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.014930 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:49.014948 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:49.018488 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":272,"threads_started":1}
W20260619 18:41:49.018968 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:49.009647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:49897 (local address 127.3.85.190:37861)
0619 18:41:49.009931 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:49.009940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:49.010116 (+ 176us) server_negotiation.cc:407] Connection header received
0619 18:41:49.010262 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:49.010266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:49.010332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:49.010429 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:49.011264 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.012062 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.013531 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.013952 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.014068 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.018727 (+ 4659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:49897: BlockingRecv error: recv got EOF from 127.3.85.130:49897 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:49.504988 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:49.496581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:34195)
0619 18:41:49.497107 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:49.497124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:49.497250 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:49.497581 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:49.497584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:49.497913 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:49.498092 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.498100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.499149 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.499153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.500076 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.500089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.500794 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.500798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.500921 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.501686 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:49.501705 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:49.504734 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":296,"threads_started":1}
W20260619 18:41:49.505301 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:49.496743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:34195 (local address 127.3.85.190:37861)
0619 18:41:49.496920 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:49.496925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:49.497211 (+ 286us) server_negotiation.cc:407] Connection header received
0619 18:41:49.497363 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:49.497367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:49.497446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:49.497535 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:49.498258 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.499005 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.500232 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.500652 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.500753 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.505020 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:34195: BlockingRecv error: recv got EOF from 127.3.85.131:34195 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:49 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:49.530187 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:49.521605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:43329)
0619 18:41:49.522118 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:49.522135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:49.522261 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:49.522725 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:49.522729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:49.523094 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:49.523243 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.523252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.524313 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.524318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.525311 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:49.525323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.526166 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.526170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:49.526308 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.527054 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:49.527072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:49.529950 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
W20260619 18:41:49.530542 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:49.521661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:43329 (local address 127.3.85.190:37861)
0619 18:41:49.521877 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:49.521881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:49.522253 (+ 372us) server_negotiation.cc:407] Connection header received
0619 18:41:49.522380 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:49.522384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:49.522517 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:49.522624 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:49.523406 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.524180 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.525492 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.525993 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.526125 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.530207 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:43329: BlockingRecv error: recv got EOF from 127.3.85.129:43329 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:50.028736 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:50.019945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:49303)
0619 18:41:50.020445 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:50.020467 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:50.020594 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:50.020999 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:50.021003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:50.021381 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:50.021524 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.021533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.022603 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.022607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.023594 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.023607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.024349 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.024354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.024505 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.025126 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:50.025149 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:50.028497 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":285,"threads_started":1}
W20260619 18:41:50.028960 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:50.020177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:49303 (local address 127.3.85.190:37861)
0619 18:41:50.020408 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:50.020416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:50.020553 (+ 137us) server_negotiation.cc:407] Connection header received
0619 18:41:50.020778 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:50.020782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:50.020851 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:50.020956 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:50.021691 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.022416 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.023757 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.024203 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.024311 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.028723 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:49303: BlockingRecv error: recv got EOF from 127.3.85.130:49303 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:50.516151 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:50.506112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:44755)
0619 18:41:50.506711 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:50.506728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:50.506855 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:50.507254 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:50.507258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:50.507670 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:50.507919 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.507933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.509558 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.509564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.510973 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.510991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.511939 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.511944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.512117 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.512845 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:50.512864 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:50.515899 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":358,"threads_started":1}
W20260619 18:41:50.516460 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:50.506230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:44755 (local address 127.3.85.190:37861)
0619 18:41:50.506554 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:50.506569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:50.506852 (+ 283us) server_negotiation.cc:407] Connection header received
0619 18:41:50.506963 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:50.506968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:50.507041 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:50.507150 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:50.508116 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.509343 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.511226 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.511736 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.511867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.516175 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:44755: BlockingRecv error: recv got EOF from 127.3.85.131:44755 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:50 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:50.540681 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:50.531362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:36535)
0619 18:41:50.531912 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:50.531929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:50.532097 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:50.532458 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:50.532463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:50.532989 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:50.533197 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.533206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.534661 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.534668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.535604 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:50.535617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.536366 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.536372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:50.536524 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.537258 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:50.537278 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:50.540317 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
W20260619 18:41:50.540789 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:50.531521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:36535 (local address 127.3.85.190:37861)
0619 18:41:50.531774 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:50.531782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:50.532035 (+ 253us) server_negotiation.cc:407] Connection header received
0619 18:41:50.532239 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:50.532252 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:50.532313 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:50.532425 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:50.533369 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.534506 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.535760 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.536187 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.536305 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.540567 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:36535: BlockingRecv error: recv got EOF from 127.3.85.129:36535 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:51.038945 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:51.029973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:39493)
0619 18:41:51.030586 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:51.030606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:51.030739 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:51.031059 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:51.031062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:51.031435 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:51.031599 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.031608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.032801 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.032808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.034024 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.034039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.034804 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.034810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.034988 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.035666 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:51.035684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:51.038577 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":296,"threads_started":1}
W20260619 18:41:51.039068 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:51.030213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:39493 (local address 127.3.85.190:37861)
0619 18:41:51.030431 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:51.030437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:51.030699 (+ 262us) server_negotiation.cc:407] Connection header received
0619 18:41:51.030849 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:51.030853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:51.030916 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:51.031016 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:51.031740 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.032637 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.034171 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.034632 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.034787 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.038790 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:39493: BlockingRecv error: recv got EOF from 127.3.85.130:39493 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:51.526307 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:51.517389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:49075)
0619 18:41:51.518019 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:51.518035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:51.518202 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:51.518575 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:51.518579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:51.519017 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:51.519176 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.519185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.520222 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.520230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.521174 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.521187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.521944 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.521950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.522133 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.522853 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:51.522872 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:51.526050 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":361,"threads_started":1}
W20260619 18:41:51.526504 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:51.517462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49075 (local address 127.3.85.190:37861)
0619 18:41:51.517802 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:51.517822 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:51.518132 (+ 310us) server_negotiation.cc:407] Connection header received
0619 18:41:51.518336 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:51.518343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:51.518411 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:51.518550 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:51.519342 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.520066 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.521335 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.521772 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.521925 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.526253 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:49075: BlockingRecv error: recv got EOF from 127.3.85.131:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:51 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:51.550292 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:51.541755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:34535)
0619 18:41:51.542260 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:51.542278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:51.542406 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:51.542772 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:51.542776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:51.543157 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:51.543321 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.543330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.544345 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.544349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.545273 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:51.545286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.546008 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.546013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:51.546136 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.546770 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:51.546788 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:51.549957 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":297,"threads_started":1}
W20260619 18:41:51.550408 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:51.541861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:34535 (local address 127.3.85.190:37861)
0619 18:41:51.542180 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:51.542189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:51.542371 (+ 182us) server_negotiation.cc:407] Connection header received
0619 18:41:51.542576 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:51.542580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:51.542644 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:51.542728 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:51.543460 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.544197 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.545432 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:51.545862 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:51.545960 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:51.550175 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:34535: BlockingRecv error: recv got EOF from 127.3.85.129:34535 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:52.054190 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:52.040111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:45621)
0619 18:41:52.040667 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:52.040689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:52.040807 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:52.041495 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:52.041500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:52.041928 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:52.042095 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.042105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.043565 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.043570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.044530 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.044544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.045388 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.045394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.045538 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.050010 (+ 4472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:52.050034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:52.053795 (+ 3761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
W20260619 18:41:52.054369 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:52.040198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45621 (local address 127.3.85.190:37861)
0619 18:41:52.040410 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:52.040417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:52.040778 (+ 361us) server_negotiation.cc:407] Connection header received
0619 18:41:52.041087 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:52.041101 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:52.041202 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:52.041381 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:52.042346 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.043378 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.044708 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.045185 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.045297 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.054094 (+ 8797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:45621: BlockingRecv error: recv got EOF from 127.3.85.130:45621 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:52.536353 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:52.527415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:48861)
0619 18:41:52.527961 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:52.527977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:52.528097 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:52.528479 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:52.528482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:52.528868 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:52.529018 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.529027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.530516 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.530521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.531476 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.531492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.532311 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.532316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.532449 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.533099 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:52.533119 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:52.536112 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
W20260619 18:41:52.536713 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:52.527483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:48861 (local address 127.3.85.190:37861)
0619 18:41:52.527674 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:52.527680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:52.528059 (+ 379us) server_negotiation.cc:407] Connection header received
0619 18:41:52.528217 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:52.528221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:52.528287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:52.528370 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:52.529272 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.530289 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.531650 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.532119 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.532241 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.536385 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:48861: BlockingRecv error: recv got EOF from 127.3.85.131:48861 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:52 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:52.560453 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:52.551582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:53703)
0619 18:41:52.552353 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:52.552373 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:52.552522 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:52.552915 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:52.552922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:52.553493 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:52.553705 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.553716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.554853 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.554859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.555797 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:52.555810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.556528 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.556533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:52.556651 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.557241 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:52.557258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:52.560189 (+ 2931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":356,"threads_started":1}
W20260619 18:41:52.560660 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:52.551692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:53703 (local address 127.3.85.190:37861)
0619 18:41:52.551894 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:52.551899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:52.552471 (+ 572us) server_negotiation.cc:407] Connection header received
0619 18:41:52.552637 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:52.552642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:52.552721 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:52.552830 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:52.553839 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.554695 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.555969 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:52.556377 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:52.556476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:52.560446 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:53703: BlockingRecv error: recv got EOF from 127.3.85.129:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:53.064635 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:53.055393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:45527)
0619 18:41:53.056039 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:53.056059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:53.056185 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:53.056652 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:53.056656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:53.057056 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:53.057238 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.057248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.058289 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.058293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.059312 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.059326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.060146 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.060151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.060276 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.060950 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:53.060978 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:53.064249 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
W20260619 18:41:53.064715 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:53.055511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45527 (local address 127.3.85.190:37861)
0619 18:41:53.055746 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:53.055754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:53.056151 (+ 397us) server_negotiation.cc:407] Connection header received
0619 18:41:53.056308 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:53.056312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:53.056394 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:53.056490 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:53.057390 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.058146 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.059511 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.059940 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.060051 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.064451 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:45527: BlockingRecv error: recv got EOF from 127.3.85.130:45527 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:53.546737 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:53.537465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:35241)
0619 18:41:53.538120 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:53.538137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:53.538333 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:53.538861 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:53.538869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:53.539433 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:53.539702 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.539718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.540988 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.540992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.541940 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.541952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.542782 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.542786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.542904 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.543491 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:53.543507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:53.546393 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":287,"threads_started":1}
W20260619 18:41:53.546823 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:53.537650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:35241 (local address 127.3.85.190:37861)
0619 18:41:53.537924 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:53.537932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:53.538236 (+ 304us) server_negotiation.cc:407] Connection header received
0619 18:41:53.538539 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:53.538544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:53.538607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:53.538735 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:53.539864 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.540841 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.542088 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.542641 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.542742 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.546651 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:35241: BlockingRecv error: recv got EOF from 127.3.85.131:35241 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:53 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:53.570389 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:53.561605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:55679)
0619 18:41:53.562285 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:53.562302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:53.562511 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:53.562884 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:53.562894 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:53.563317 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:53.563522 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.563534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.564613 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.564619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.565566 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:53.565579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.566360 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.566368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:53.566562 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.567234 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:53.567252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:53.570153 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":437,"threads_started":1}
W20260619 18:41:53.570770 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:53.561870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:55679 (local address 127.3.85.190:37861)
0619 18:41:53.562143 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:53.562149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:53.562402 (+ 253us) server_negotiation.cc:407] Connection header received
0619 18:41:53.562649 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:53.562654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:53.562746 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:53.562854 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:53.563644 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.564470 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.565720 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.566157 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.566293 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.570425 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:55679: BlockingRecv error: recv got EOF from 127.3.85.129:55679 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:54.074613 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:54.065692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:42743)
0619 18:41:54.066231 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:54.066251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:54.066413 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:54.066870 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:54.066875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:54.067294 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:54.067435 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.067444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.068757 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.068764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.069672 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.069683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.070432 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.070437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.070641 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.071301 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:54.071318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:54.074222 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":273,"threads_started":1}
W20260619 18:41:54.074888 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:54.065778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:42743 (local address 127.3.85.190:37861)
0619 18:41:54.066026 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:54.066034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:54.066340 (+ 306us) server_negotiation.cc:407] Connection header received
0619 18:41:54.066604 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:54.066608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:54.066712 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:54.066844 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:54.067594 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.068555 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.069829 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.070238 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.070384 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.074517 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:42743: BlockingRecv error: recv got EOF from 127.3.85.130:42743 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:54.558188 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:54.548337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:52975)
0619 18:41:54.548931 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:54.548954 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:54.549079 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:54.549377 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:54.549380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:54.549771 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:54.549926 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.549934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.551497 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.551503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.552655 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.552683 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.553716 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.553721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.553857 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.554572 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:54.554609 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:54.557841 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":291,"threads_started":1}
W20260619 18:41:54.558367 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:54.548419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:52975 (local address 127.3.85.190:37861)
0619 18:41:54.548614 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:54.548618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:54.549093 (+ 475us) server_negotiation.cc:407] Connection header received
0619 18:41:54.549162 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:54.549166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:54.549239 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:54.549327 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:54.550241 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.551212 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.552948 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.553477 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.553602 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.558087 (+ 4485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:52975: BlockingRecv error: recv got EOF from 127.3.85.131:52975 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:54 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:54.581321 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:54.571693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:57473)
0619 18:41:54.572297 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:54.572314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:54.572599 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:54.572976 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:54.572981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:54.573370 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:54.573527 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.573536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.574845 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.574851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.576042 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:54.576058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.576819 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.576824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:54.576997 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.577751 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:54.577782 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:54.580951 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":404,"threads_started":1}
W20260619 18:41:54.581441 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:54.571774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:57473 (local address 127.3.85.190:37861)
0619 18:41:54.572014 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:54.572021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:54.572409 (+ 388us) server_negotiation.cc:407] Connection header received
0619 18:41:54.572749 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:54.572758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:54.572817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:54.572935 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:54.573672 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.574670 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.576205 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.576631 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.576769 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.581201 (+ 4432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:57473: BlockingRecv error: recv got EOF from 127.3.85.129:57473 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:55.084584 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:55.076075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:47795)
0619 18:41:55.076557 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:55.076578 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:55.076696 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:55.077008 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:55.077011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:55.077385 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:55.077548 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.077556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.078599 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.078602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.079637 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.079650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.080433 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.080439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.080573 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.081268 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:55.081286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:55.084240 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":251,"threads_started":1}
W20260619 18:41:55.084728 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:55.076154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:47795 (local address 127.3.85.190:37861)
0619 18:41:55.076379 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:55.076385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:55.076662 (+ 277us) server_negotiation.cc:407] Connection header received
0619 18:41:55.076802 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:55.076805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:55.076866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:55.076946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:55.077681 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.078439 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.079770 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.080266 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.080364 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.084487 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:47795: BlockingRecv error: recv got EOF from 127.3.85.130:47795 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:55.568740 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:55.559479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:59477)
0619 18:41:55.560037 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:55.560055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:55.560175 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:55.560480 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:55.560483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:55.560895 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:55.561063 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.561072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.562211 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.562216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.563309 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.563324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.564080 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.564084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.564224 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.564913 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:55.564933 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:55.568419 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
W20260619 18:41:55.568890 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:55.559627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:59477 (local address 127.3.85.190:37861)
0619 18:41:55.559816 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:55.559820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:55.560137 (+ 317us) server_negotiation.cc:407] Connection header received
0619 18:41:55.560278 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:55.560282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:55.560343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:55.560433 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:55.561247 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.562038 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.563474 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.563919 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.564032 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.568662 (+ 4630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:59477: BlockingRecv error: recv got EOF from 127.3.85.131:59477 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:55 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:55.591120 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:55.582354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:45997)
0619 18:41:55.582856 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:55.582873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:55.582988 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:55.583306 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:55.583309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:55.583678 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:55.583826 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.583835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.584968 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.584973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.585994 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:55.586010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.586871 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.586876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:55.587032 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.587629 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:55.587646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:55.590768 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":310,"threads_started":1}
W20260619 18:41:55.591280 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:55.582418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:45997 (local address 127.3.85.190:37861)
0619 18:41:55.582697 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:55.582703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:55.582955 (+ 252us) server_negotiation.cc:407] Connection header received
0619 18:41:55.583114 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:55.583118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:55.583177 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:55.583283 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:55.583961 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.584839 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.586184 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:55.586696 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:55.586804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:55.590992 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:45997: BlockingRecv error: recv got EOF from 127.3.85.129:45997 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:56.094542 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:56.085595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:35255)
0619 18:41:56.086127 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:56.086149 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:56.086310 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:56.086671 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:56.086675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:56.087116 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:56.087269 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.087277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.088390 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.088396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.089413 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.089426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.090225 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.090232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.090415 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.091239 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:56.091257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:56.094180 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":319,"threads_started":1}
W20260619 18:41:56.094707 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:56.085680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:35255 (local address 127.3.85.190:37861)
0619 18:41:56.085908 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:56.085917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:56.086243 (+ 326us) server_negotiation.cc:407] Connection header received
0619 18:41:56.086471 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:56.086475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:56.086535 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:56.086643 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:56.087425 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.088239 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.089562 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.090051 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.090188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.094417 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:35255: BlockingRecv error: recv got EOF from 127.3.85.130:35255 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:56.579984 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:56.569917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:38891)
0619 18:41:56.570509 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:56.570548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:56.570744 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:56.571290 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:56.571293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:56.571659 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:56.571808 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.571817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.573032 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.573037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.574181 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.574214 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.575307 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.575315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.575498 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.576414 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:56.576433 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:56.579749 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":312,"threads_started":1}
W20260619 18:41:56.580286 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:56.570377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:38891 (local address 127.3.85.190:37861)
0619 18:41:56.570617 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:56.570621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:56.570640 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:56.571005 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:56.571010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:56.571073 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:56.571178 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:56.571980 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.572842 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.574422 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.575074 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.575229 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.579984 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:38891: BlockingRecv error: recv got EOF from 127.3.85.131:38891 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:56 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:56.601826 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:56.592346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:42277)
0619 18:41:56.592890 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:56.592907 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:56.593080 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:56.593451 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:56.593456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:56.593896 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:56.594065 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.594074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.595606 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.595611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.596640 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:56.596654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.597434 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.597439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:56.597561 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.598186 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:56.598205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:56.601526 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
W20260619 18:41:56.601990 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:56.592459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:42277 (local address 127.3.85.190:37861)
0619 18:41:56.592721 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:56.592728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:56.593006 (+ 278us) server_negotiation.cc:407] Connection header received
0619 18:41:56.593224 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:56.593228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:56.593305 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:56.593421 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:56.594216 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.595411 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.596801 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:56.597286 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:56.597397 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:56.601738 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:42277: BlockingRecv error: recv got EOF from 127.3.85.129:42277 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:57.108279 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:57.095692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:53481)
0619 18:41:57.096172 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:57.096193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:57.096382 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:57.096742 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:57.096747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:57.097172 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:57.097351 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.097360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.098380 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.098387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.099321 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.099333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.100097 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.100103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.100289 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.104655 (+ 4366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:57.104687 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:57.108023 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
W20260619 18:41:57.108536 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:57.095717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:53481 (local address 127.3.85.190:37861)
0619 18:41:57.095944 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:57.095951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:57.096300 (+ 349us) server_negotiation.cc:407] Connection header received
0619 18:41:57.096509 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:57.096515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:57.096582 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:57.096686 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:57.097507 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.098239 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.099475 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.099934 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.100074 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.108255 (+ 8181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:53481: BlockingRecv error: recv got EOF from 127.3.85.130:53481 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:57.591014 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:57.581249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:39117)
0619 18:41:57.581836 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:57.581854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:57.581988 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:57.582357 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:57.582360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:57.582896 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:57.583137 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.583152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.584867 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.584875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.585887 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.585901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.586681 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.586686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.586839 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.587458 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:57.587476 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:57.590630 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":313,"threads_started":1}
W20260619 18:41:57.591087 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:57.581339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:39117 (local address 127.3.85.190:37861)
0619 18:41:57.581590 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:57.581594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:57.581956 (+ 362us) server_negotiation.cc:407] Connection header received
0619 18:41:57.582095 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:57.582099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:57.582173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:57.582259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:57.583303 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.584457 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.586041 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.586528 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.586645 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.590857 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:39117: BlockingRecv error: recv got EOF from 127.3.85.131:39117 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:57 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:57.611711 12700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:57.602968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:36951)
0619 18:41:57.603550 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:57.603568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:57.603693 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:57.604051 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:57.604055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:57.604462 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:57.604617 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.604625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.605638 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.605641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.606662 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:57.606675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.607429 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.607433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:57.607560 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.608250 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:57.608268 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:57.611388 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":329,"threads_started":1}
W20260619 18:41:57.611781 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:57.603089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:36951 (local address 127.3.85.190:37861)
0619 18:41:57.603259 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:57.603263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:57.603652 (+ 389us) server_negotiation.cc:407] Connection header received
0619 18:41:57.603793 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:57.603797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:57.603904 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:57.603998 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:57.604766 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.605506 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.606792 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.607288 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.607389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.611555 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:36951: BlockingRecv error: recv got EOF from 127.3.85.129:36951 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:58.118012 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:58.109341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:55517)
0619 18:41:58.109889 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:58.109908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:58.110084 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:58.110472 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:58.110477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:58.110880 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:58.111032 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.111042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.112047 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.112054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.112978 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.112992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.113772 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.113778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.113966 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.114681 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:58.114702 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:58.117770 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
W20260619 18:41:58.118352 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:58.109424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:55517 (local address 127.3.85.190:37861)
0619 18:41:58.109651 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:58.109660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:58.110005 (+ 345us) server_negotiation.cc:407] Connection header received
0619 18:41:58.110212 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:58.110217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:58.110298 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:58.110406 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:58.111186 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.111892 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.113141 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.113591 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.113742 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.118069 (+ 4327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:55517: BlockingRecv error: recv got EOF from 127.3.85.130:55517 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:58.601224 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:58.592186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:39781)
0619 18:41:58.592766 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:58.592783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:58.592965 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:58.593372 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:58.593376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:58.593799 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:58.593961 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.593970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.595156 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.595163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.596183 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.596197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.597001 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.597008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.597192 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.597852 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:58.597870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:58.600876 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":326,"threads_started":1}
W20260619 18:41:58.601339 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:58.592267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:39781 (local address 127.3.85.190:37861)
0619 18:41:58.592564 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:58.592574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:58.592889 (+ 315us) server_negotiation.cc:407] Connection header received
0619 18:41:58.593105 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:58.593112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:58.593181 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:58.593296 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:58.594124 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.594994 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.596347 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.596829 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.596980 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.601097 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:39781: BlockingRecv error: recv got EOF from 127.3.85.131:39781 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:58 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:58.621767 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:58.612742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:60497)
0619 18:41:58.613252 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:58.613269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:58.613397 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:58.613768 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:58.613771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:58.614127 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:58.614300 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.614308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.615325 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.615330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.616255 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:58.616269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.617170 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.617175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:58.617391 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.618259 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:58.618282 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:58.621489 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":252,"threads_started":1}
W20260619 18:41:58.621944 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:58.612733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:60497 (local address 127.3.85.190:37861)
0619 18:41:58.612951 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:58.612960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:58.613379 (+ 419us) server_negotiation.cc:407] Connection header received
0619 18:41:58.613510 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:58.613519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:58.613625 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:58.613736 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:58.614426 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.615187 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.616438 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.616973 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.617115 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.621716 (+ 4601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:60497: BlockingRecv error: recv got EOF from 127.3.85.129:60497 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:59.128044 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:59.119188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:59073)
0619 18:41:59.119791 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:59.119811 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:59.119971 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:59.120355 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:59.120360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:59.120783 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:59.120953 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.120962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.122023 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.122029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.123145 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.123158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.123898 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.123905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.124067 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.124716 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:59.124746 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:59.127808 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
W20260619 18:41:59.128233 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:59.119452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:59073 (local address 127.3.85.190:37861)
0619 18:41:59.119706 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:59.119713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:59.119897 (+ 184us) server_negotiation.cc:407] Connection header received
0619 18:41:59.120129 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:59.120133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:59.120195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:59.120323 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:59.121133 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.121875 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.123295 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.123710 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.123857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.128006 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:59073: BlockingRecv error: recv got EOF from 127.3.85.130:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:59.611279 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:59.602593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:45863)
0619 18:41:59.603148 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:59.603164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:59.603316 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:59.603721 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:59.603726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:59.604129 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:59.604359 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.604370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.605363 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.605368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.606304 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.606315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.607092 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.607097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.607213 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.607785 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:59.607803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:59.611019 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":292,"threads_started":1}
W20260619 18:41:59.611512 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:59.602666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:45863 (local address 127.3.85.190:37861)
0619 18:41:59.602871 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:59.602877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:59.603230 (+ 353us) server_negotiation.cc:407] Connection header received
0619 18:41:59.603469 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:59.603474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:59.603552 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:59.603662 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:59.604488 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.605219 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.606500 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.606917 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.607016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.611237 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:45863: BlockingRecv error: recv got EOF from 127.3.85.131:45863 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:41:59 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:41:59.631623 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:59.622888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:37759)
0619 18:41:59.623398 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:59.623417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:59.623545 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:59.623890 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:59.623894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:59.624260 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:59.624441 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.624451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.625471 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.625474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.626396 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:59.626408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.627190 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.627195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:59.627317 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.628092 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:59.628115 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:59.631367 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":312,"threads_started":1}
W20260619 18:41:59.631834 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:59.622989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:37759 (local address 127.3.85.190:37861)
0619 18:41:59.623244 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:59.623248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:59.623507 (+ 259us) server_negotiation.cc:407] Connection header received
0619 18:41:59.623667 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:59.623671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:59.623751 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:59.623846 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:59.624583 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.625338 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.626604 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:59.627021 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:59.627177 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:59.631574 (+ 4397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:37759: BlockingRecv error: recv got EOF from 127.3.85.129:37759 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:42:00.138517 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:00.129218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:44343)
0619 18:42:00.130105 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:00.130141 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:00.130263 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:00.130712 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:00.130715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:00.131200 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:00.131447 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.131457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.132812 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.132817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.133756 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.133769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.134535 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.134539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.134670 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.135271 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:00.135290 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:00.138191 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":583,"threads_started":1}
W20260619 18:42:00.138823 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:00.129415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44343 (local address 127.3.85.190:37861)
0619 18:42:00.129687 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:00.129692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:00.130240 (+ 548us) server_negotiation.cc:407] Connection header received
0619 18:42:00.130381 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:00.130389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:00.130525 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:00.130641 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:00.131614 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.132663 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.133923 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.134367 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.134570 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.138577 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:44343: BlockingRecv error: recv got EOF from 127.3.85.130:44343 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.131@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:42:00.621470 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:00.612274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.131:49479)
0619 18:42:00.612775 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:00.612797 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:00.612912 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:00.613259 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:00.613263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:00.613670 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:00.613850 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.613859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.615047 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.615052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.616013 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.616026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.617078 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.617085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.617405 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.618056 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:00.618074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:00.621103 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":281,"threads_started":1}
W20260619 18:42:00.621626 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:00.612335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49479 (local address 127.3.85.190:37861)
0619 18:42:00.612522 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:00.612527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:00.612876 (+ 349us) server_negotiation.cc:407] Connection header received
0619 18:42:00.613032 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:00.613036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:00.613115 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:00.613203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:00.614022 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.614867 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.616197 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.616876 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.617045 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.621375 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.131:49479: BlockingRecv error: recv got EOF from 127.3.85.131:49479 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:42:00 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.129@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:42:00.642619 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:00.632951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.129:38543)
0619 18:42:00.633790 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:00.633822 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:00.634143 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:00.634661 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:00.634668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:00.635248 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:00.635534 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.635548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.636801 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.636805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.637745 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:00.637757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.638570 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.638578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:00.638758 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.639389 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:00.639407 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:00.642292 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
W20260619 18:42:00.642877 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:00.633103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:38543 (local address 127.3.85.190:37861)
0619 18:42:00.633341 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:00.633349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:00.634042 (+ 693us) server_negotiation.cc:407] Connection header received
0619 18:42:00.634305 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:00.634310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:00.634430 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:00.634580 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:00.635727 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.636651 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.637894 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:00.638333 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:00.638434 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:00.642588 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.129:38543: BlockingRecv error: recv got EOF from 127.3.85.129:38543 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 18:42:01 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:42:01 dist-test-slave-09fp krb5kdc[11663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.85.130@KRBTEST.COM for oryx/127.3.85.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:42:01.148793 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.139867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37861 (local address 127.3.85.130:52753)
0619 18:42:01.140412 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:01.140431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:01.140584 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:01.140957 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:01.140962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:01.141382 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:01.141574 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:01.141583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.142691 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.142698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:01.143626 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:01.143639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.144459 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.144466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:01.144634 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.145340 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:01.145358 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:01.148334 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.85.190:37861: Server oryx/127.3.85.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":320,"threads_started":1}
W20260619 18:42:01.148996 12401 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:01.139981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:52753 (local address 127.3.85.190:37861)
0619 18:42:01.140239 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.140253 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.140514 (+ 261us) server_negotiation.cc:407] Connection header received
0619 18:42:01.140712 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.140718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.140790 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.140896 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.141720 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.142511 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.143787 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.144292 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.144443 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.148693 (+ 4250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.85.130:52753: BlockingRecv error: recv got EOF from 127.3.85.130:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:42:01.451903 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12252
I20260619 18:42:01.465641 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.453170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.3.85.129:35113)
0619 18:42:01.453793 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.453800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.453828 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:42:01.453891 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.453897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.454150 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.454317 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.456001 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.457114 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.458753 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.459389 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.459517 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.462230 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:01.462240 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:01.462256 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:01.462295 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:01.465078 (+ 2783us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:42:01.465387 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52154: Request ticket server kudu/127.3.85.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":336,"threads_started":1}
W20260619 18:42:01.465734 12712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52154: Request ticket server kudu/127.3.85.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:42:01.466064 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12252: Not authorized: Client connection negotiation failed: client connection to 127.3.85.129:35113: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:42:01.466301 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12252
I20260619 18:42:01.484002 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12403
I20260619 18:42:01.497036 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.485086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.3.85.130:36665)
0619 18:42:01.485677 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.485684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.485714 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:42:01.485774 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.485780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.486017 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.486218 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.487638 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.488831 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.490084 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.490776 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.490932 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.493557 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:01.493567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:01.493583 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:01.493625 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:01.496294 (+ 2669us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:42:01.496687 (+ 393us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40526: Request ticket server kudu/127.3.85.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":270,"threads_started":1}
W20260619 18:42:01.497159 12713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40526: Request ticket server kudu/127.3.85.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:42:01.497149 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12403: Not authorized: Client connection negotiation failed: client connection to 127.3.85.130:36665: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:42:01.497288 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12403
I20260619 18:42:01.514158 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12534
I20260619 18:42:01.527488 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.515397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.3.85.131:34977)
0619 18:42:01.516025 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.516035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.516062 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:42:01.516124 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.516131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.516370 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.516557 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.517980 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.519266 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.520559 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.521107 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.521212 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.523954 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:01.523963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:01.523979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:01.524024 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:01.526909 (+ 2885us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:42:01.527224 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46758: Request ticket server kudu/127.3.85.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":372,"threads_started":1}
W20260619 18:42:01.527616 12714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46758: Request ticket server kudu/127.3.85.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:42:01.527660 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12534: Not authorized: Client connection negotiation failed: client connection to 127.3.85.131:34977: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.85.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:42:01.527848 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12534
I20260619 18:42:01.544523 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12183
I20260619 18:42:01.545905 12245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:01.660815 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12183
2026-06-19T18:42:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21454 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T18:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:01Z Disabled control of system clock
I20260619 18:42:01.743722 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43323
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:42:02.129370 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:02.129705 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:02.129850 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:02.137334 12719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:02.137473 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:02.137562 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:02.137651 12719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:02.137737 12719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:02.147359 12719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:43323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:02.148916 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:02.150841 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:02.163991 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.163897 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.163969 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:02.165294 12719 server_base.cc:1034] running on GCE node
I20260619 18:42:02.166203 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:02.167879 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:02.169137 12719 hybrid_clock.cc:648] HybridClock initialized: now 1781894522169091 us; error 34 us; skew 500 ppm
I20260619 18:42:02.169612 12719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:02.176456 12719 webserver.cc:460] Webserver started at http://127.3.85.190:44227/ using document root <none> and password file <none>
I20260619 18:42:02.177354 12719 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:02.177476 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:02.177860 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:02.181227 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "09ee9e73f59a4386989e527762b1ed83"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "b8e849d6b62b18a93f4e906bdca6500e"
server_key_iv: "ccb5b16911648bf56e8ef5c1b902f271"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.182132 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "09ee9e73f59a4386989e527762b1ed83"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "b8e849d6b62b18a93f4e906bdca6500e"
server_key_iv: "ccb5b16911648bf56e8ef5c1b902f271"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.188114 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:42:02.192488 12733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:02.194175 12719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 18:42:02.194439 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "09ee9e73f59a4386989e527762b1ed83"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "b8e849d6b62b18a93f4e906bdca6500e"
server_key_iv: "ccb5b16911648bf56e8ef5c1b902f271"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.194770 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:02.213337 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:02.214299 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:02.214658 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:02.234414 12719 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:43323
I20260619 18:42:02.234539 12784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:43323 every 8 connection(s)
I20260619 18:42:02.236019 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:02.240278 12785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:02.240514 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12719
I20260619 18:42:02.240696 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:42:02.241366 3414 external_mini_cluster.cc:1428] Setting key 92c263fc9c0132831564ba41f68c7a24
I20260619 18:42:02.255743 12785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83: Bootstrap starting.
I20260619 18:42:02.258708 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:02.243620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.3.85.190:43323)
0619 18:42:02.244817 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:02.244840 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:02.244897 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:42:02.247465 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:02.247504 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:02.248283 (+ 779us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:02.248831 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:02.249404 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.251388 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.252931 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.254101 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.254319 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:02.255591 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:02.255602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:02.255623 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:02.256384 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:02.256490 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:02.256518 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:02.256639 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:02.256882 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 18:42:02.257237 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":343,"threads_started":1}
I20260619 18:42:02.259896 12785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:02.261056 12785 log.cc:824] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:02.264509 12785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83: No bootstrap required, opened a new log
I20260619 18:42:02.272616 12785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } }
I20260619 18:42:02.273097 12785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:02.273229 12785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ee9e73f59a4386989e527762b1ed83, State: Initialized, Role: FOLLOWER
I20260619 18:42:02.273872 12785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } }
I20260619 18:42:02.274142 12785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:02.274291 12785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:02.274518 12785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:02.276916 12785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } }
I20260619 18:42:02.277555 12785 leader_election.cc:302] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09ee9e73f59a4386989e527762b1ed83; no voters:
I20260619 18:42:02.278329 12785 leader_election.cc:288] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:02.278539 12790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:02.279847 12790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [term 1 LEADER]: Becoming Leader. State: Replica: 09ee9e73f59a4386989e527762b1ed83, State: Running, Role: LEADER
I20260619 18:42:02.280474 12790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } }
I20260619 18:42:02.280956 12785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:02.283638 12791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09ee9e73f59a4386989e527762b1ed83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } } }
I20260619 18:42:02.283758 12792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09ee9e73f59a4386989e527762b1ed83. Latest consensus state: current_term: 1 leader_uuid: "09ee9e73f59a4386989e527762b1ed83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ee9e73f59a4386989e527762b1ed83" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 43323 } } }
I20260619 18:42:02.284188 12791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:02.284257 12792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:02.285144 12799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:02.291242 12799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:02.300603 12799 catalog_manager.cc:1347] Generated new cluster ID: ddbeb507383e48f58a851369003fee7d
I20260619 18:42:02.300737 12799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:02.321806 12799 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:02.323330 12799 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:02.329270 12799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09ee9e73f59a4386989e527762b1ed83: Generated new TSK 0
I20260619 18:42:02.330272 12799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:02.335734 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--builtin_ntp_servers=127.3.85.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 18:42:02.690428 12809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:02.690834 12809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:02.691015 12809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:02.698971 12809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:02.699090 12809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:02.699257 12809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:42:02.710834 12809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:02.712684 12809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:02.715325 12809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:02.731115 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.731154 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.731084 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:02.732168 12809 server_base.cc:1034] running on GCE node
I20260619 18:42:02.732920 12809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:02.734009 12809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:02.735234 12809 hybrid_clock.cc:648] HybridClock initialized: now 1781894522735149 us; error 90 us; skew 500 ppm
I20260619 18:42:02.735622 12809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:02.738268 12809 webserver.cc:460] Webserver started at http://127.3.85.129:40295/ using document root <none> and password file <none>
I20260619 18:42:02.739097 12809 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:02.739236 12809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:02.739596 12809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:02.742789 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "27989d21093b41309e0adf52dcd6981b"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "01011fdc075f956784710a29b6afa4a1"
server_key_iv: "3bf3404deda0ff548578c9a1a6031196"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.743695 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "27989d21093b41309e0adf52dcd6981b"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "01011fdc075f956784710a29b6afa4a1"
server_key_iv: "3bf3404deda0ff548578c9a1a6031196"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.749775 12809 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:42:02.753809 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:02.755439 12809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:02.755662 12809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "27989d21093b41309e0adf52dcd6981b"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-09fp"
server_key: "01011fdc075f956784710a29b6afa4a1"
server_key_iv: "3bf3404deda0ff548578c9a1a6031196"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.755924 12809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:02.791230 12809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:02.792194 12809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:02.792542 12809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:02.793568 12809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:02.795322 12809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:02.795461 12809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:02.795608 12809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:02.795696 12809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:02.830129 12809 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:42809
I20260619 18:42:02.830188 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:42809 every 8 connection(s)
I20260619 18:42:02.831602 12809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:02.836561 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12809
I20260619 18:42:02.836776 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:02.837600 3414 external_mini_cluster.cc:1428] Setting key 2b2b35f62d75bf4dae5b20039c858e8b
I20260619 18:42:02.840965 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--builtin_ntp_servers=127.3.85.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:42:02.847709 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:02.834547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:42519 (local address 127.3.85.190:43323)
0619 18:42:02.835016 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:02.835021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:02.836481 (+ 1460us) server_negotiation.cc:407] Connection header received
0619 18:42:02.839112 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:02.839119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:02.839198 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:02.839316 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:02.840861 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.841699 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.844434 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.844860 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.844978 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:02.846853 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:02.846858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:02.846862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:02.846883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:02.846924 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:02.846936 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:02.847023 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:02.847352 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 18:42:02.847442 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":248,"threads_started":1}
I20260619 18:42:02.849077 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:02.835140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43323 (local address 127.3.85.129:42519)
0619 18:42:02.836236 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:02.836293 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:02.838785 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:02.839590 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:02.839604 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:02.839739 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:02.840627 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:02.840654 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.841889 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.841897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:02.844210 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:02.844231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.845127 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.845136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:02.845508 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:02.846389 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:02.846395 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:02.846488 (+ 93us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:02.846608 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:02.847051 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:02.847064 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:02.847075 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:42:02.847271 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 18:42:02.847747 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":313,"threads_started":1}
I20260619 18:42:02.850916 12936 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43323
I20260619 18:42:02.851253 12936 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:02.852092 12936 heartbeater.cc:511] Master 127.3.85.190:43323 requested a full tablet report, sending...
I20260619 18:42:02.854444 12750 ts_manager.cc:194] Registered new tserver with Master: 27989d21093b41309e0adf52dcd6981b (127.3.85.129:42809)
I20260619 18:42:02.856106 12750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.129:42519
W20260619 18:42:03.198700 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:03.199121 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:03.199271 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:03.207193 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:03.207363 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:03.207541 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:42:03.218053 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:03.219619 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:03.221530 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:03.236330 12948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:03.236268 12945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:03.237134 12946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:03.239404 12940 server_base.cc:1034] running on GCE node
I20260619 18:42:03.240231 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:03.241448 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:03.242690 12940 hybrid_clock.cc:648] HybridClock initialized: now 1781894523242636 us; error 54 us; skew 500 ppm
I20260619 18:42:03.243075 12940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:03.246021 12940 webserver.cc:460] Webserver started at http://127.3.85.130:34485/ using document root <none> and password file <none>
I20260619 18:42:03.247028 12940 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:03.247133 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:03.247534 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:03.250663 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "17598846f3144ffaa5d6ea7b5798db65"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "13e6f327543da1bc3ec8f1c6079036cf"
server_key_iv: "24703812a7919297b5ad5871275a5dbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.251583 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "17598846f3144ffaa5d6ea7b5798db65"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "13e6f327543da1bc3ec8f1c6079036cf"
server_key_iv: "24703812a7919297b5ad5871275a5dbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.257282 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:42:03.261370 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.263031 12940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:42:03.263236 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "17598846f3144ffaa5d6ea7b5798db65"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "13e6f327543da1bc3ec8f1c6079036cf"
server_key_iv: "24703812a7919297b5ad5871275a5dbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.263501 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:03.289680 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:03.290678 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:03.291021 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:03.292091 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:03.293712 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:03.293807 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.293972 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:03.294066 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.328929 12940 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:37137
I20260619 18:42:03.329034 13066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:37137 every 8 connection(s)
I20260619 18:42:03.330407 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:03.338546 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 12940
I20260619 18:42:03.338716 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:03.339162 3414 external_mini_cluster.cc:1428] Setting key 39ccd90d7e178b9614e2dbec2dba1ce5
I20260619 18:42:03.342438 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--builtin_ntp_servers=127.3.85.148:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:42:03.344556 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.333161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44177 (local address 127.3.85.190:43323)
0619 18:42:03.333371 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.333375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.334920 (+ 1545us) server_negotiation.cc:407] Connection header received
0619 18:42:03.336747 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.336752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.336828 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.336951 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.338161 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.339258 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.341453 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.341848 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.341951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.343774 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.343779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.343782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.343801 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.343834 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.343848 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.343940 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.344226 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.344314 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:42:03.346212 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.333683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43323 (local address 127.3.85.130:44177)
0619 18:42:03.334747 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:03.334793 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:03.336426 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:03.337174 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:03.337182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:03.337290 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:03.337919 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:03.337940 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.339454 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.339461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:03.341267 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:03.341284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.342117 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.342128 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:03.342522 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.343327 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:03.343333 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:03.343398 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:03.343537 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:03.343976 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:03.343991 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:03.344002 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:42:03.344210 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 18:42:03.344714 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":300,"threads_started":1}
I20260619 18:42:03.347934 13067 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43323
I20260619 18:42:03.348300 13067 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:03.349126 13067 heartbeater.cc:511] Master 127.3.85.190:43323 requested a full tablet report, sending...
I20260619 18:42:03.351024 12750 ts_manager.cc:194] Registered new tserver with Master: 17598846f3144ffaa5d6ea7b5798db65 (127.3.85.130:37137)
I20260619 18:42:03.351900 12750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.130:44177
W20260619 18:42:03.691514 13070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:03.691812 13070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:03.691957 13070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:03.699620 13070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:03.699738 13070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:03.699939 13070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:42:03.709498 13070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:43323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:03.710918 13070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:03.712805 13070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:03.727350 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:03.727377 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:03.727519 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:03.728832 13070 server_base.cc:1034] running on GCE node
I20260619 18:42:03.729604 13070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:03.730751 13070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:03.732021 13070 hybrid_clock.cc:648] HybridClock initialized: now 1781894523731976 us; error 67 us; skew 500 ppm
I20260619 18:42:03.732400 13070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:03.735121 13070 webserver.cc:460] Webserver started at http://127.3.85.131:44199/ using document root <none> and password file <none>
I20260619 18:42:03.735929 13070 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:03.736075 13070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:03.736423 13070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:03.739568 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "7a40473a3f374bbab49d400b71e8c203"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "c413554d251f6358c60579d540b392b1"
server_key_iv: "c262e8f286fed5bca81d3c6d7ab91d6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.740504 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "7a40473a3f374bbab49d400b71e8c203"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "c413554d251f6358c60579d540b392b1"
server_key_iv: "c262e8f286fed5bca81d3c6d7ab91d6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.746176 13070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:42:03.750146 13084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.751845 13070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:42:03.752063 13070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "7a40473a3f374bbab49d400b71e8c203"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-09fp"
server_key: "c413554d251f6358c60579d540b392b1"
server_key_iv: "c262e8f286fed5bca81d3c6d7ab91d6c"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.752329 13070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:03.779923 13070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:03.780874 13070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:03.781229 13070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:03.782438 13070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:03.784107 13070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:03.784248 13070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.784387 13070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:03.784478 13070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.820717 13070 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46285
I20260619 18:42:03.820777 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46285 every 8 connection(s)
I20260619 18:42:03.822045 13070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:03.829941 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13070
I20260619 18:42:03.830123 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:03.830742 3414 external_mini_cluster.cc:1428] Setting key ee397f670f354972ec2f53ff6a99b89b
I20260619 18:42:03.836870 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.824855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49921 (local address 127.3.85.190:43323)
0619 18:42:03.825078 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.825082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.826798 (+ 1716us) server_negotiation.cc:407] Connection header received
0619 18:42:03.828608 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.828613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.828688 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.828789 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.830244 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.831401 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.833667 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.834308 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.834436 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.836077 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.836083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.836086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.836104 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.836135 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.836148 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.836224 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.836495 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.836583 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 18:42:03.837899 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.825402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:43323 (local address 127.3.85.131:49921)
0619 18:42:03.826573 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:03.826630 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:03.828259 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:03.829038 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:03.829047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:03.829154 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:03.829977 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:03.830007 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.831600 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.831605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:03.833497 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:03.833513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.834737 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.834743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:03.834971 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.835735 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:03.835737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:03.835777 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:03.835864 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:03.836239 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:03.836245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:03.836251 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:42:03.836435 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 18:42:03.836767 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
I20260619 18:42:03.839401 13197 heartbeater.cc:347] Connected to a master server at 127.3.85.190:43323
I20260619 18:42:03.839684 13197 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:03.840449 13197 heartbeater.cc:511] Master 127.3.85.190:43323 requested a full tablet report, sending...
I20260619 18:42:03.841964 12750 ts_manager.cc:194] Registered new tserver with Master: 7a40473a3f374bbab49d400b71e8c203 (127.3.85.131:46285)
I20260619 18:42:03.842834 12750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.131:49921
I20260619 18:42:03.847139 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:42:03.853374 12938 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:03.851764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.3.85.190:43323)
0619 18:42:03.851968 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.851976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.852396 (+ 420us) server_negotiation.cc:407] Connection header received
0619 18:42:03.852577 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.852581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.852652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.852760 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.853179 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49492: BlockingRecv error: recv got EOF from 127.0.0.1:49492 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:42:03.854394 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12809
I20260619 18:42:03.859957 12936 heartbeater.cc:503] Master 127.3.85.190:43323 was elected leader, sending a full tablet report...
I20260619 18:42:03.863099 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.855367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.3.85.129:42809)
0619 18:42:03.856022 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.856029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.856057 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:42:03.856117 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.856124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.856419 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.856556 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.856915 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.857993 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.859310 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.860038 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.860204 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.861504 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.861513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.861552 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.862210 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.862261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.862281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.862426 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.862624 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.862760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":363,"threads_started":1}
I20260619 18:42:03.864447 12931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:03.949429 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12809
I20260619 18:42:03.966990 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12940
I20260619 18:42:03.975400 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.968039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.3.85.130:37137)
0619 18:42:03.968653 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.968660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.968687 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:42:03.968746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.968752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.968979 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.969151 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.969582 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.970664 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.971924 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:03.972499 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:03.972654 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:03.973759 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.973767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.973781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.974393 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.974441 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.974598 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.974724 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.974902 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.975036 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260619 18:42:03.976635 13062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:04.060513 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12940
I20260619 18:42:04.077001 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13070
I20260619 18:42:04.085935 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:04.078235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.3.85.131:46285)
0619 18:42:04.078952 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:04.078960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:04.078997 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:42:04.079058 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:04.079064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:04.079306 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:04.079458 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:04.079862 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:04.080926 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:04.082327 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:04.083048 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:04.083251 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:04.084385 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:04.084394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:04.084408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:04.085089 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:04.085139 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:04.085185 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:04.085284 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:04.085454 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:42:04.085607 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":413,"threads_started":1}
I20260619 18:42:04.087385 13192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:04.170244 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13070
I20260619 18:42:04.186748 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 12719
I20260619 18:42:04.187996 12780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:04.289556 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 12719
2026-06-19T18:42:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2618 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T18:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:04Z Disabled control of system clock
I20260619 18:42:04.363016 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40643
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:42:04.721012 13216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:04.721352 13216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:04.721439 13216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:04.729115 13216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:04.729249 13216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:04.729287 13216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:04.729346 13216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:04.729390 13216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:04.739328 13216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:40643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:04.740644 13216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:04.742481 13216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:04.755141 13222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:04.755062 13224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:04.755827 13216 server_base.cc:1034] running on GCE node
W20260619 18:42:04.755970 13221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:04.756898 13216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:04.758613 13216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:04.759902 13216 hybrid_clock.cc:648] HybridClock initialized: now 1781894524759857 us; error 63 us; skew 500 ppm
I20260619 18:42:04.760322 13216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:04.762997 13216 webserver.cc:460] Webserver started at http://127.3.85.190:46723/ using document root <none> and password file <none>
I20260619 18:42:04.763819 13216 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:04.763964 13216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:04.764312 13216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:04.769296 13216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "c0bc651d5a0942bfac7cd06fff1b24fc"
format_stamp: "Formatted at 2026-06-19 18:42:04 on dist-test-slave-09fp"
server_key: "367fedb990f1359697b3c8e85df128f2"
server_key_iv: "538c01d695cf74e8a0465f9d27d8e135"
server_key_version: "encryptionkey@0"
I20260619 18:42:04.770139 13216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "c0bc651d5a0942bfac7cd06fff1b24fc"
format_stamp: "Formatted at 2026-06-19 18:42:04 on dist-test-slave-09fp"
server_key: "367fedb990f1359697b3c8e85df128f2"
server_key_iv: "538c01d695cf74e8a0465f9d27d8e135"
server_key_version: "encryptionkey@0"
I20260619 18:42:04.775923 13216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:42:04.780336 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:04.781991 13216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:42:04.782223 13216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "c0bc651d5a0942bfac7cd06fff1b24fc"
format_stamp: "Formatted at 2026-06-19 18:42:04 on dist-test-slave-09fp"
server_key: "367fedb990f1359697b3c8e85df128f2"
server_key_iv: "538c01d695cf74e8a0465f9d27d8e135"
server_key_version: "encryptionkey@0"
I20260619 18:42:04.782539 13216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:04.788966 13216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:04.789892 13216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:04.790232 13216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:04.810316 13216 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:40643
I20260619 18:42:04.810355 13281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:40643 every 8 connection(s)
I20260619 18:42:04.811880 13216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:04.816206 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:04.819077 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13216
I20260619 18:42:04.819325 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:42:04.819983 3414 external_mini_cluster.cc:1428] Setting key 1c55c793badb1fbcbd99e2c277db02d8
I20260619 18:42:04.828984 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:04.822053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.3.85.190:40643)
0619 18:42:04.823251 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:04.823264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:04.823311 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:42:04.825044 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:04.825071 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:04.825603 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:04.826005 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:04.827225 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:04.827232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:04.827246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:04.827265 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:04.827326 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:04.827342 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:04.827539 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:04.827679 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:42:04.828014 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":458,"threads_started":1}
I20260619 18:42:04.829759 13282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc: Bootstrap starting.
I20260619 18:42:04.833266 13282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:04.834380 13282 log.cc:824] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:04.837884 13282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc: No bootstrap required, opened a new log
I20260619 18:42:04.845849 13282 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } }
I20260619 18:42:04.846246 13282 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:04.846321 13282 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bc651d5a0942bfac7cd06fff1b24fc, State: Initialized, Role: FOLLOWER
I20260619 18:42:04.846946 13282 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } }
I20260619 18:42:04.847160 13282 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:04.847259 13282 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:04.847388 13282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:04.849654 13282 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } }
I20260619 18:42:04.850354 13282 leader_election.cc:302] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0bc651d5a0942bfac7cd06fff1b24fc; no voters:
I20260619 18:42:04.851183 13282 leader_election.cc:288] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:04.851295 13287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:04.852087 13287 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [term 1 LEADER]: Becoming Leader. State: Replica: c0bc651d5a0942bfac7cd06fff1b24fc, State: Running, Role: LEADER
I20260619 18:42:04.852706 13287 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } }
I20260619 18:42:04.853444 13282 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:04.855775 13289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bc651d5a0942bfac7cd06fff1b24fc. Latest consensus state: current_term: 1 leader_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } } }
I20260619 18:42:04.855803 13288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc651d5a0942bfac7cd06fff1b24fc" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 40643 } } }
I20260619 18:42:04.856873 13289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:04.856909 13288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:04.863955 13294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:04.869481 13294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:04.878746 13294 catalog_manager.cc:1347] Generated new cluster ID: b7a064f35b874281a24f13b95dee6321
I20260619 18:42:04.878918 13294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:04.901589 13294 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:04.902093 13294 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:04.910657 13294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0bc651d5a0942bfac7cd06fff1b24fc: Generated new TSK 0
I20260619 18:42:04.911818 13294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:04.932597 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--builtin_ntp_servers=127.3.85.148:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:42:05.294271 13306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:05.294673 13306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:05.294835 13306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:05.302778 13306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:05.302909 13306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:05.303108 13306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:42:05.313498 13306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:05.314982 13306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:05.316893 13306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:05.331570 13311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.331533 13314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.331511 13312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.332304 13306 server_base.cc:1034] running on GCE node
I20260619 18:42:05.332948 13306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:05.333931 13306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:05.335161 13306 hybrid_clock.cc:648] HybridClock initialized: now 1781894525335085 us; error 68 us; skew 500 ppm
I20260619 18:42:05.335526 13306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:05.338534 13306 webserver.cc:460] Webserver started at http://127.3.85.129:43275/ using document root <none> and password file <none>
I20260619 18:42:05.339442 13306 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:05.339541 13306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:05.339896 13306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:05.343078 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "637e4b89492f4b01afd71bd43c09efd4"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "e834addedc6973fc067fa86db971d0a5"
server_key_iv: "0e79d470c2b8cd8a095a44e11fa4076f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.343973 13306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "637e4b89492f4b01afd71bd43c09efd4"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "e834addedc6973fc067fa86db971d0a5"
server_key_iv: "0e79d470c2b8cd8a095a44e11fa4076f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.349877 13306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:42:05.353994 13320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.355569 13306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:05.355790 13306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "637e4b89492f4b01afd71bd43c09efd4"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "e834addedc6973fc067fa86db971d0a5"
server_key_iv: "0e79d470c2b8cd8a095a44e11fa4076f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.356041 13306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:05.362747 13306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:05.363658 13306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:05.363961 13306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:05.365005 13306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:05.366619 13306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:05.366757 13306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.366904 13306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:05.366992 13306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.402411 13306 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:44811
I20260619 18:42:05.402539 13432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:44811 every 8 connection(s)
I20260619 18:42:05.403863 13306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:05.410830 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13306
I20260619 18:42:05.411051 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:05.411667 3414 external_mini_cluster.cc:1428] Setting key c21e87f4f64359d62c558247935bfa8f
I20260619 18:42:05.412963 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.406574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:47691 (local address 127.3.85.190:40643)
0619 18:42:05.407038 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.407043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.408343 (+ 1300us) server_negotiation.cc:407] Connection header received
0619 18:42:05.410150 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.410156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.410260 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.410403 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.412215 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.412224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.412229 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:05.412259 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.412302 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:05.412315 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.412425 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.412680 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.412745 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":257,"threads_started":1}
I20260619 18:42:05.414310 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.407102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40643 (local address 127.3.85.129:47691)
0619 18:42:05.408135 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.408196 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.409827 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.410804 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.410820 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.410995 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.411864 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:05.411867 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.411910 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:05.412010 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.412433 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.412447 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.412456 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.412752 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.413052 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":329,"threads_started":1}
I20260619 18:42:05.414743 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--builtin_ntp_servers=127.3.85.148:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:42:05.416390 13433 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40643
I20260619 18:42:05.416790 13433 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:05.417510 13433 heartbeater.cc:511] Master 127.3.85.190:40643 requested a full tablet report, sending...
I20260619 18:42:05.420219 13247 ts_manager.cc:194] Registered new tserver with Master: 637e4b89492f4b01afd71bd43c09efd4 (127.3.85.129:44811)
W20260619 18:42:05.779874 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:05.780260 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:05.780433 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:05.788472 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:05.788636 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:05.788822 13437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:42:05.798795 13437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:05.800251 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:05.802199 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:05.817087 13445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.817108 13443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.817235 13442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.818522 13437 server_base.cc:1034] running on GCE node
I20260619 18:42:05.819324 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:05.820629 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:05.821926 13437 hybrid_clock.cc:648] HybridClock initialized: now 1781894525821855 us; error 69 us; skew 500 ppm
I20260619 18:42:05.822338 13437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:05.824959 13437 webserver.cc:460] Webserver started at http://127.3.85.130:34459/ using document root <none> and password file <none>
I20260619 18:42:05.825786 13437 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:05.825934 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:05.826292 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:05.829321 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "a8621ab4c0f043078692e3a18f85b690"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "b61b57577493e5dd68d9603d4b607e48"
server_key_iv: "f807d917dd7c6cfcfe0381eebfee1e19"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.830220 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "a8621ab4c0f043078692e3a18f85b690"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "b61b57577493e5dd68d9603d4b607e48"
server_key_iv: "f807d917dd7c6cfcfe0381eebfee1e19"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.836043 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:42:05.840047 13451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.841630 13437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:05.841849 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "a8621ab4c0f043078692e3a18f85b690"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-09fp"
server_key: "b61b57577493e5dd68d9603d4b607e48"
server_key_iv: "f807d917dd7c6cfcfe0381eebfee1e19"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.842134 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:05.848474 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:05.849364 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:05.849653 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:05.850692 13437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:05.852324 13437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:05.852463 13437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.852609 13437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:05.852703 13437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.888707 13437 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:46473
I20260619 18:42:05.888758 13563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:46473 every 8 connection(s)
I20260619 18:42:05.890120 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:05.891893 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13437
I20260619 18:42:05.892087 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:05.892521 3414 external_mini_cluster.cc:1428] Setting key 9c317d7d5eb9cff742f34a17614a5462
I20260619 18:42:05.895395 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--builtin_ntp_servers=127.3.85.148:43825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:42:05.899696 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.893157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:57301 (local address 127.3.85.190:40643)
0619 18:42:05.893366 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.893374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.895143 (+ 1769us) server_negotiation.cc:407] Connection header received
0619 18:42:05.896977 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.896986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.897074 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.897192 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.898849 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.898858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.898865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:05.898891 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.898936 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:05.898955 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.899111 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.899269 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.899339 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:42:05.900660 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.893739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40643 (local address 127.3.85.130:57301)
0619 18:42:05.894951 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.895006 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.896653 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.897452 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.897461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.897568 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.898440 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:05.898486 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.898541 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:05.898643 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.899063 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.899070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.899076 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.899312 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.899543 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":415,"threads_started":1}
I20260619 18:42:05.902539 13564 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40643
I20260619 18:42:05.902890 13564 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:05.903568 13564 heartbeater.cc:511] Master 127.3.85.190:40643 requested a full tablet report, sending...
I20260619 18:42:05.905258 13247 ts_manager.cc:194] Registered new tserver with Master: a8621ab4c0f043078692e3a18f85b690 (127.3.85.130:46473)
W20260619 18:42:06.253046 13567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:06.253347 13567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:06.253499 13567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:06.261390 13567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:06.261511 13567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:06.261711 13567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:42:06.272678 13567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:40643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:06.274075 13567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:06.275957 13567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:06.289691 13572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:06.289896 13575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:06.289916 13573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:06.291021 13567 server_base.cc:1034] running on GCE node
I20260619 18:42:06.291687 13567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:06.292744 13567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:06.293990 13567 hybrid_clock.cc:648] HybridClock initialized: now 1781894526293940 us; error 46 us; skew 500 ppm
I20260619 18:42:06.294341 13567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:42:06.297219 13567 webserver.cc:460] Webserver started at http://127.3.85.131:40857/ using document root <none> and password file <none>
I20260619 18:42:06.298004 13567 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:06.298153 13567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:06.298569 13567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:06.301611 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "c1e83bc383de41428dcc5dafd4def0b7"
format_stamp: "Formatted at 2026-06-19 18:42:06 on dist-test-slave-09fp"
server_key: "c9e9e7279f4493ad3ac526a7d47cf3e2"
server_key_iv: "7b880c3196b1880453257a983f333cd3"
server_key_version: "encryptionkey@0"
I20260619 18:42:06.302541 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "c1e83bc383de41428dcc5dafd4def0b7"
format_stamp: "Formatted at 2026-06-19 18:42:06 on dist-test-slave-09fp"
server_key: "c9e9e7279f4493ad3ac526a7d47cf3e2"
server_key_iv: "7b880c3196b1880453257a983f333cd3"
server_key_version: "encryptionkey@0"
I20260619 18:42:06.308094 13567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:42:06.312037 13581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:06.313598 13567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:06.313824 13567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "c1e83bc383de41428dcc5dafd4def0b7"
format_stamp: "Formatted at 2026-06-19 18:42:06 on dist-test-slave-09fp"
server_key: "c9e9e7279f4493ad3ac526a7d47cf3e2"
server_key_iv: "7b880c3196b1880453257a983f333cd3"
server_key_version: "encryptionkey@0"
I20260619 18:42:06.314077 13567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:06.320230 13567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:06.321169 13567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:06.321481 13567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:06.322464 13567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:06.323976 13567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:06.324111 13567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:06.324265 13567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:06.324355 13567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:06.361204 13567 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:43937
I20260619 18:42:06.361259 13693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:43937 every 8 connection(s)
I20260619 18:42:06.362726 13567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:06.371322 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.365525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:38893 (local address 127.3.85.190:40643)
0619 18:42:06.365729 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.365735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.367316 (+ 1581us) server_negotiation.cc:407] Connection header received
0619 18:42:06.368989 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.368993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.369063 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.369182 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.370675 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:06.370680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:06.370683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:06.370702 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:06.370735 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:06.370748 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:06.370860 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.371037 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.371091 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:42:06.372304 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.366076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:40643 (local address 127.3.85.131:38893)
0619 18:42:06.367156 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:06.367195 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:06.368707 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:06.369398 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:06.369406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:06.369507 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:06.370259 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:42:06.370261 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:06.370303 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:42:06.370398 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:06.370848 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:06.370855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:06.370861 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:42:06.371101 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 18:42:06.371318 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":407,"threads_started":1}
I20260619 18:42:06.372334 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13567
I20260619 18:42:06.372526 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:06.373003 3414 external_mini_cluster.cc:1428] Setting key e3c3cd0db56eb98710ef0c8dfe56d9c8
I20260619 18:42:06.374091 13694 heartbeater.cc:347] Connected to a master server at 127.3.85.190:40643
I20260619 18:42:06.374521 13694 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:06.375267 13694 heartbeater.cc:511] Master 127.3.85.190:40643 requested a full tablet report, sending...
I20260619 18:42:06.376849 13247 ts_manager.cc:194] Registered new tserver with Master: c1e83bc383de41428dcc5dafd4def0b7 (127.3.85.131:43937)
I20260619 18:42:06.389098 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:42:06.395854 13435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:06.394011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.3.85.190:40643)
0619 18:42:06.394183 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.394188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.394832 (+ 644us) server_negotiation.cc:407] Connection header received
0619 18:42:06.394996 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.395001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.395058 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.395159 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.395530 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55160: BlockingRecv error: recv got EOF from 127.0.0.1:55160 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:06.396667 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13306
I20260619 18:42:06.400749 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.397725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.3.85.129:44811)
0619 18:42:06.398257 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.398265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.398295 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:42:06.398358 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.398365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.398764 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.398907 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.399891 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:06.399898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:06.399910 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:06.399923 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:06.399969 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:06.399987 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:06.400191 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.400403 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.400499 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260619 18:42:06.402356 13428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:06.486255 13433 heartbeater.cc:503] Master 127.3.85.190:40643 was elected leader, sending a full tablet report...
I20260619 18:42:06.487587 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13306
W20260619 18:42:06.504356 13232 connection.cc:535] server connection from 127.3.85.129:47691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:42:06.504712 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13437
I20260619 18:42:06.508414 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.505713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.3.85.130:46473)
0619 18:42:06.506250 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.506257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.506286 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:42:06.506348 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.506354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.506655 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.506791 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.507697 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:06.507703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:06.507716 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:06.507732 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:06.507770 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:06.507784 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:06.507893 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.507980 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.508096 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":277,"threads_started":1}
I20260619 18:42:06.509699 13559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:06.593128 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13437
I20260619 18:42:06.609364 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13567
I20260619 18:42:06.613723 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.610496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.3.85.131:43937)
0619 18:42:06.611158 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.611165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.611196 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:42:06.611259 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.611266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.611485 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.611619 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.612919 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:06.612928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:06.612951 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:06.612973 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:06.613020 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:06.613035 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:06.613233 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.613311 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.613415 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":358,"threads_started":1}
I20260619 18:42:06.615098 13689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:06.697796 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13567
I20260619 18:42:06.716914 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13216
I20260619 18:42:06.718129 13277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:06.818933 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13216
2026-06-19T18:42:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2532 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:06 dist-test-slave-09fp krb5kdc[13712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:06 dist-test-slave-09fp krb5kdc[13712](info): set up 2 sockets
Jun 19 18:42:06 dist-test-slave-09fp krb5kdc[13712](info): commencing operation
krb5kdc: starting...
W20260619 18:42:07.939702 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:42:08 dist-test-slave-09fp krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:42:08.195528 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38197
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:08.535591 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:08.535984 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:08.536157 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:08.543668 13725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:08.543865 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:08.543977 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:08.544082 13725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:08.544174 13725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:08.554394 13725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:38197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:08.555886 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:08.557291 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:08.570830 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.570868 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.570909 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:08.571436 13725 server_base.cc:1034] running on GCE node
I20260619 18:42:08.572309 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:08.573896 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:08.575151 13725 hybrid_clock.cc:648] HybridClock initialized: now 1781894528575105 us; error 47 us; skew 500 ppm
I20260619 18:42:08.575523 13725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:08 dist-test-slave-09fp krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:08.586184 13725 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:42:08.588004 13725 webserver.cc:460] Webserver started at http://127.3.85.190:45947/ using document root <none> and password file <none>
I20260619 18:42:08.588805 13725 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:08.588965 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:08.589284 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:08.592532 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "28fb37e2206147a2a0cf6130d60507be"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-09fp"
server_key: "85f196784f5fa13c3e614eaa209e713b"
server_key_iv: "71d598ac4c5161ef7f602ae44c243216"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.593436 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "28fb37e2206147a2a0cf6130d60507be"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-09fp"
server_key: "85f196784f5fa13c3e614eaa209e713b"
server_key_iv: "71d598ac4c5161ef7f602ae44c243216"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.599680 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:42:08.603713 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:08.605269 13725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:42:08.605491 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "28fb37e2206147a2a0cf6130d60507be"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-09fp"
server_key: "85f196784f5fa13c3e614eaa209e713b"
server_key_iv: "71d598ac4c5161ef7f602ae44c243216"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.605722 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:08.629743 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:08.634565 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:08.634887 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:08.654927 13725 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:38197
I20260619 18:42:08.654978 13791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:38197 every 8 connection(s)
I20260619 18:42:08.656438 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:08.660630 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13725
I20260619 18:42:08.660737 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:08.660914 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:42:08.661695 3414 external_mini_cluster.cc:1428] Setting key afdbbc5265758b16144b64800ab45b11
I20260619 18:42:08.675974 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: Bootstrap starting.
Jun 19 18:42:08 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:42:08.680533 13792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:08.681770 13792 log.cc:824] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:08.685498 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: No bootstrap required, opened a new log
I20260619 18:42:08.688551 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.663989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.3.85.190:38197)
0619 18:42:08.664901 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.664918 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.664965 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:42:08.667329 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.667360 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.667869 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.668272 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.670250 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.671727 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.673723 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.674634 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.674805 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.680172 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.680181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.680200 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.680250 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.683734 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.684788 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.684796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.684803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.684994 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.685439 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.685442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.685444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.686075 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.686338 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.686918 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.687365 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":428,"threads_started":1}
I20260619 18:42:08.695235 13792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } }
I20260619 18:42:08.695719 13792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:08.695832 13792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fb37e2206147a2a0cf6130d60507be, State: Initialized, Role: FOLLOWER
I20260619 18:42:08.696537 13792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } }
I20260619 18:42:08.696799 13792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:08.696960 13792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:08.697167 13792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:08.699759 13792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } }
I20260619 18:42:08.700284 13792 leader_election.cc:302] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28fb37e2206147a2a0cf6130d60507be; no voters:
I20260619 18:42:08.700927 13792 leader_election.cc:288] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:08.701109 13797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:08.701807 13797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [term 1 LEADER]: Becoming Leader. State: Replica: 28fb37e2206147a2a0cf6130d60507be, State: Running, Role: LEADER
I20260619 18:42:08.702379 13797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } }
I20260619 18:42:08.703222 13792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:08.710214 13798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28fb37e2206147a2a0cf6130d60507be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } } }
I20260619 18:42:08.711413 13798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:08.709529 13799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28fb37e2206147a2a0cf6130d60507be. Latest consensus state: current_term: 1 leader_uuid: "28fb37e2206147a2a0cf6130d60507be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28fb37e2206147a2a0cf6130d60507be" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38197 } } }
I20260619 18:42:08.713299 13799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:08.714747 13804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:08.720981 13804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:08.730567 13804 catalog_manager.cc:1347] Generated new cluster ID: 5236ab9d3e49413683da43a163f2f934
I20260619 18:42:08.730715 13804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:08.744320 13804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:08.745982 13804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:08.754133 13804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: Generated new TSK 0
I20260619 18:42:08.755142 13804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:42:08.830396 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:09.196563 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:09.196898 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:09.197000 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:09.204941 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:09.205075 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:09.205192 13818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:42:09.215876 13818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:09.217182 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:09.218650 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:09.232046 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:09.232146 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:09.233202 13826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:09.234158 13818 server_base.cc:1034] running on GCE node
I20260619 18:42:09.234923 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:09.236001 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:09.237251 13818 hybrid_clock.cc:648] HybridClock initialized: now 1781894529237189 us; error 68 us; skew 500 ppm
I20260619 18:42:09.237601 13818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:09 dist-test-slave-09fp krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:09.241173 13818 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:42:09.242908 13818 webserver.cc:460] Webserver started at http://127.3.85.129:40823/ using document root <none> and password file <none>
I20260619 18:42:09.243616 13818 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:09.243721 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:09.243983 13818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:09.247180 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "e458f45881bc4b75b8d9b775c605669f"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "659b5461e64bd3f6c08a30b38b064f8c"
server_key_iv: "751a1bf0e60431e4a5a946301f281290"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.247979 13818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "e458f45881bc4b75b8d9b775c605669f"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "659b5461e64bd3f6c08a30b38b064f8c"
server_key_iv: "751a1bf0e60431e4a5a946301f281290"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.253700 13818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:42:09.257931 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.259809 13818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:42:09.260033 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "e458f45881bc4b75b8d9b775c605669f"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "659b5461e64bd3f6c08a30b38b064f8c"
server_key_iv: "751a1bf0e60431e4a5a946301f281290"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.260239 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:09.278393 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:09.283231 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:09.283562 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:09.284626 13818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:09.286096 13818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:09.286173 13818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.286334 13818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:09.286401 13818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.321400 13818 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:38711
I20260619 18:42:09.321435 13945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:38711 every 8 connection(s)
I20260619 18:42:09.322783 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:09.330433 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13818
I20260619 18:42:09.330693 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:09.331512 3414 external_mini_cluster.cc:1428] Setting key 4fb17e4bcc61f9dceaa01a99a12c65a6
Jun 19 18:42:09 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:42:09.346997 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.325532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:33455 (local address 127.3.85.190:38197)
0619 18:42:09.325978 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.325984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.327147 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 18:42:09.329031 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.329038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.329135 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.329271 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:09.331475 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.332392 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.334770 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.335179 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.335288 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.340201 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:09.340212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:09.340220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:09.340276 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:09.343532 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:09.344450 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:09.344463 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:09.344466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:09.344598 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:09.345099 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:09.345104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:09.345107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:09.345392 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:09.345558 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.346326 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.346633 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260619 18:42:09.348068 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.326150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38197 (local address 127.3.85.129:33455)
0619 18:42:09.326969 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:09.327009 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:09.328730 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:09.329532 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:09.329541 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:09.330272 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:09.331220 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:09.331253 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.332584 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.332590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:09.334563 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:09.334586 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.335455 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.335465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:09.335834 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.336667 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:09.336707 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:09.339847 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:09.343724 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:09.343733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:09.343783 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:09.344279 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:09.344729 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:09.344736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:09.344744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:09.344952 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:09.345532 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:09.345540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:09.345928 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 18:42:09.346225 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 18:42:09.346825 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":378,"threads_started":1}
I20260619 18:42:09.349709 13946 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38197
I20260619 18:42:09.350055 13946 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:09.350909 13946 heartbeater.cc:511] Master 127.3.85.190:38197 requested a full tablet report, sending...
I20260619 18:42:09.353250 13757 ts_manager.cc:194] Registered new tserver with Master: e458f45881bc4b75b8d9b775c605669f (127.3.85.129:38711)
I20260619 18:42:09.354853 13757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:33455
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:42:09.390589 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:09.742784 13952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:09.743139 13952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:09.743240 13952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:09.750782 13952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:09.750906 13952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:09.751022 13952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:42:09.761155 13952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:09.762493 13952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:09.763795 13952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:09.778021 13957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:09.777944 13960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:09.777980 13958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:09.779013 13952 server_base.cc:1034] running on GCE node
I20260619 18:42:09.779690 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:09.780750 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:09.781975 13952 hybrid_clock.cc:648] HybridClock initialized: now 1781894529781927 us; error 41 us; skew 500 ppm
I20260619 18:42:09.782328 13952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:09 dist-test-slave-09fp krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:09.785727 13952 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:42:09.787521 13952 webserver.cc:460] Webserver started at http://127.3.85.130:43737/ using document root <none> and password file <none>
I20260619 18:42:09.788302 13952 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:09.788466 13952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:09.788805 13952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:09.792093 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "b6be7d8d8a3b40c8ac6748fde887ac88"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "525b071b508cddf7013165cbe8e803ec"
server_key_iv: "eca17cea5503fb261c587d26f39d2c98"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.792995 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "b6be7d8d8a3b40c8ac6748fde887ac88"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "525b071b508cddf7013165cbe8e803ec"
server_key_iv: "eca17cea5503fb261c587d26f39d2c98"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.798936 13952 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:42:09.802955 13967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.804648 13952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:42:09.804867 13952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "b6be7d8d8a3b40c8ac6748fde887ac88"
format_stamp: "Formatted at 2026-06-19 18:42:09 on dist-test-slave-09fp"
server_key: "525b071b508cddf7013165cbe8e803ec"
server_key_iv: "eca17cea5503fb261c587d26f39d2c98"
server_key_version: "encryptionkey@0"
I20260619 18:42:09.805100 13952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:09.826076 13952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:09.830542 13952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:09.830866 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:09.831933 13952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:09.833276 13952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:09.833382 13952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.833529 13952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:09.833600 13952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:09.870363 13952 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:41689
I20260619 18:42:09.870406 14079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:41689 every 8 connection(s)
I20260619 18:42:09.871874 13952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:42:09.875344 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 13952
I20260619 18:42:09.875581 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:42:09.876140 3414 external_mini_cluster.cc:1428] Setting key 78712d317aa6f7dd2b1b4fe1c2c229c6
Jun 19 18:42:09 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894529, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:42:09.898407 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.874698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:35157 (local address 127.3.85.190:38197)
0619 18:42:09.875289 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.875297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.876492 (+ 1195us) server_negotiation.cc:407] Connection header received
0619 18:42:09.878874 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.878880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.878954 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.879149 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:09.881579 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.882363 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.885191 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.885662 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.885815 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.890235 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:09.890245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:09.890253 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:09.890305 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:09.894288 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:09.895355 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:09.895364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:09.895370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:09.895544 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:09.896068 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:09.896077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:09.896083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:09.896502 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:09.896663 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.897587 (+ 924us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.897874 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":319,"threads_started":1}
I20260619 18:42:09.899251 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.875352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38197 (local address 127.3.85.130:35157)
0619 18:42:09.876280 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:09.876343 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:09.878497 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:09.879486 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:09.879503 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:09.880386 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:09.881319 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:09.881349 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.882608 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.882617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:09.884949 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:09.884974 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.885959 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.885969 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:09.886344 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.887250 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:09.887281 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:09.889859 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:09.894597 (+ 4738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:09.894606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:09.894623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:09.895177 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:09.895697 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:09.895701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:09.895703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:09.895917 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:09.896734 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:09.896742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:09.897148 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 18:42:09.897425 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 18:42:09.897865 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":377,"threads_started":1}
I20260619 18:42:09.901096 14080 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38197
I20260619 18:42:09.901487 14080 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:09.902592 14080 heartbeater.cc:511] Master 127.3.85.190:38197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:42:09.904639 13757 ts_manager.cc:194] Registered new tserver with Master: b6be7d8d8a3b40c8ac6748fde887ac88 (127.3.85.130:41689)
I20260619 18:42:09.905665 13757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:35157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:42:09.935837 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:10.281031 14086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:10.281342 14086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:10.281522 14086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:10.289136 14086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:10.289266 14086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:10.289456 14086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:42:10.299603 14086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:10.301041 14086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:10.302516 14086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:10.316851 14091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:10.316818 14094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:10.316793 14092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:10.317667 14086 server_base.cc:1034] running on GCE node
I20260619 18:42:10.318374 14086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:10.319514 14086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:10.320756 14086 hybrid_clock.cc:648] HybridClock initialized: now 1781894530320711 us; error 48 us; skew 500 ppm
I20260619 18:42:10.321143 14086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:10 dist-test-slave-09fp krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894530, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:10.328308 14086 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:42:10.330046 14086 webserver.cc:460] Webserver started at http://127.3.85.131:33513/ using document root <none> and password file <none>
I20260619 18:42:10.330880 14086 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:10.331035 14086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:10.331367 14086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:10.334663 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "f874daa72d6645d69024872c9ac4fe06"
format_stamp: "Formatted at 2026-06-19 18:42:10 on dist-test-slave-09fp"
server_key: "bc2a16348701f28e423c9064ce35b28f"
server_key_iv: "240945f5f924f8eea30945c739063aa5"
server_key_version: "encryptionkey@0"
I20260619 18:42:10.335584 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "f874daa72d6645d69024872c9ac4fe06"
format_stamp: "Formatted at 2026-06-19 18:42:10 on dist-test-slave-09fp"
server_key: "bc2a16348701f28e423c9064ce35b28f"
server_key_iv: "240945f5f924f8eea30945c739063aa5"
server_key_version: "encryptionkey@0"
I20260619 18:42:10.341620 14086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:42:10.345784 14101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:10.347455 14086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:10.347672 14086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "f874daa72d6645d69024872c9ac4fe06"
format_stamp: "Formatted at 2026-06-19 18:42:10 on dist-test-slave-09fp"
server_key: "bc2a16348701f28e423c9064ce35b28f"
server_key_iv: "240945f5f924f8eea30945c739063aa5"
server_key_version: "encryptionkey@0"
I20260619 18:42:10.347911 14086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:10.358250 13946 heartbeater.cc:503] Master 127.3.85.190:38197 was elected leader, sending a full tablet report...
I20260619 18:42:10.373103 14086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:10.377918 14086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:10.378183 14086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:10.379196 14086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:10.380599 14086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:10.380679 14086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:10.380777 14086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:10.380823 14086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:10.417213 14086 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:34045
I20260619 18:42:10.417308 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:34045 every 8 connection(s)
I20260619 18:42:10.418720 14086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:42:10.420653 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 14086
I20260619 18:42:10.420889 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:42:10.421455 3414 external_mini_cluster.cc:1428] Setting key 96003c1ead2bd8a46816ba4ee41f98a5
Jun 19 18:42:10 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894530, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:42:10.440946 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.421470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:40615 (local address 127.3.85.190:38197)
0619 18:42:10.421943 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.421948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.423321 (+ 1373us) server_negotiation.cc:407] Connection header received
0619 18:42:10.425653 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.425659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.425773 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.425895 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:10.427606 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.428387 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.430040 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.430519 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.430650 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.434570 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:10.434576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:10.434579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:10.434616 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:10.437957 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.438871 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.438876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.438878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.439005 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.439422 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.439425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.439427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.439689 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:10.439803 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.440496 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.440664 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260619 18:42:10.441995 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.422142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38197 (local address 127.3.85.131:40615)
0619 18:42:10.423155 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.423203 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.425367 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.426159 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.426168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.426794 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:10.427420 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.427439 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.428571 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.428576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.429886 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.429898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.430766 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.430773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.431023 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.431873 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:10.431898 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:10.434269 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:10.438113 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:10.438121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:10.438137 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:10.438743 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:10.439106 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:10.439109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:10.439111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:10.439306 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:10.439795 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:10.439802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:10.440157 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.440398 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.440806 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":395,"threads_started":1}
I20260619 18:42:10.443482 14214 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38197
I20260619 18:42:10.443825 14214 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:10.444614 14214 heartbeater.cc:511] Master 127.3.85.190:38197 requested a full tablet report, sending...
I20260619 18:42:10.446282 13757 ts_manager.cc:194] Registered new tserver with Master: f874daa72d6645d69024872c9ac4fe06 (127.3.85.131:34045)
I20260619 18:42:10.447161 13757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:40615
I20260619 18:42:10.450201 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:42:10.468207 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.455060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.3.85.190:38197)
0619 18:42:10.455315 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.455326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.455739 (+ 413us) server_negotiation.cc:407] Connection header received
0619 18:42:10.455854 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.455858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.455941 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.456034 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:10.457413 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.458151 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.459722 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.460119 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.460232 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.462725 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:10.462731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:10.462734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:10.462769 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:10.465466 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.466232 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.466236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.466238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.466370 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.466810 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.466813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.466815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.467154 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:10.467353 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.467666 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.467843 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:42:10.473073 13757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:42:10.480679 13757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:42:10.507504 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.496682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:38711 (local address 127.0.0.1:46790)
0619 18:42:10.498353 (+ 1671us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.498371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.498513 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.499164 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.499172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.499209 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:10.499622 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.499636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.502100 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.502105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.504512 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.504527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.506553 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.506559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.506802 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.506874 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.507009 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.507194 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1547,"mutex_wait_us":62,"thread_start_us":478,"threads_started":1}
I20260619 18:42:10.508498 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.497221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.3.85.129:38711)
0619 18:42:10.497867 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.497877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.498635 (+ 758us) server_negotiation.cc:407] Connection header received
0619 18:42:10.498726 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.498732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.498949 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.499161 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:10.499850 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.501926 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.504696 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.505951 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.506129 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.506397 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.507942 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.508151 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":281,"threads_started":1}
I20260619 18:42:10.508689 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.494515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.3.85.130:41689)
0619 18:42:10.500259 (+ 5744us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.500265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.500289 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:42:10.500428 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.500436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.500655 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.500925 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:10.501628 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.503206 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.505920 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.507478 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.507648 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.507889 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.508128 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.508291 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5593,"thread_start_us":5464,"threads_started":1}
I20260619 18:42:10.508647 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.494316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:41689 (local address 127.0.0.1:50888)
0619 18:42:10.495134 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.495177 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.495942 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.500953 (+ 5011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.500959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.500993 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:10.501442 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.501458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.503373 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.503379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.505730 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.505739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.507740 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.507746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.507936 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.507967 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.508062 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.508198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":29,"thread_start_us":363,"threads_started":1}
I20260619 18:42:10.508745 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.495685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:34045 (local address 127.0.0.1:50328)
0619 18:42:10.496591 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.496608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.496714 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.500582 (+ 3868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.500587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.500615 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:10.501062 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.501077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.503026 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.503031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.505683 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.505702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.507797 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.507803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.507995 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.508019 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.508115 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.508292 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":84,"thread_start_us":611,"threads_started":1}
I20260619 18:42:10.510651 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.495975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.3.85.131:34045)
0619 18:42:10.499917 (+ 3942us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.499929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.499967 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:42:10.500069 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.500080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.500354 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.500564 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:10.501310 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.502848 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.506134 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.507557 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.508919 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.509205 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.509278 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.509516 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3761,"thread_start_us":3587,"threads_started":1}
I20260619 18:42:10.513149 14015 tablet_service.cc:1467] Processing CreateTablet for tablet a5a943b7b35d4554932f259757689832 (DEFAULT_TABLE table=test-table [id=79893394b7c14e99a091ce5f593019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:10.514588 14015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a943b7b35d4554932f259757689832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:10.514487 14149 tablet_service.cc:1467] Processing CreateTablet for tablet a5a943b7b35d4554932f259757689832 (DEFAULT_TABLE table=test-table [id=79893394b7c14e99a091ce5f593019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:10.514484 13881 tablet_service.cc:1467] Processing CreateTablet for tablet a5a943b7b35d4554932f259757689832 (DEFAULT_TABLE table=test-table [id=79893394b7c14e99a091ce5f593019f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:42:10.515969 13881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a943b7b35d4554932f259757689832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:10.515985 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a943b7b35d4554932f259757689832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:10.526949 14233 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Bootstrap starting.
I20260619 18:42:10.530305 14233 tablet_bootstrap.cc:654] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:10.531389 14234 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Bootstrap starting.
I20260619 18:42:10.531765 14233 log.cc:824] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:10.533111 14235 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Bootstrap starting.
I20260619 18:42:10.536209 14234 tablet_bootstrap.cc:654] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:10.536651 14235 tablet_bootstrap.cc:654] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:10.537549 14234 log.cc:824] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:10.537635 14235 log.cc:824] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:10.537739 14233 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: No bootstrap required, opened a new log
I20260619 18:42:10.538091 14233 ts_tablet_manager.cc:1397] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 18:42:10.540925 14235 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: No bootstrap required, opened a new log
I20260619 18:42:10.540926 14234 tablet_bootstrap.cc:492] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: No bootstrap required, opened a new log
I20260619 18:42:10.541215 14235 ts_tablet_manager.cc:1397] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 18:42:10.541230 14234 ts_tablet_manager.cc:1397] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:42:10.546396 14233 raft_consensus.cc:348] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.546908 14233 raft_consensus.cc:374] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:10.547052 14233 raft_consensus.cc:729] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6be7d8d8a3b40c8ac6748fde887ac88, State: Initialized, Role: FOLLOWER
I20260619 18:42:10.547736 14233 consensus_queue.cc:260] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.548947 14080 heartbeater.cc:503] Master 127.3.85.190:38197 was elected leader, sending a full tablet report...
I20260619 18:42:10.549705 14233 ts_tablet_manager.cc:1428] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Time spent starting tablet: real 0.011s user 0.002s sys 0.008s
I20260619 18:42:10.552646 14234 raft_consensus.cc:348] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.552646 14235 raft_consensus.cc:348] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.553200 14235 raft_consensus.cc:374] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:10.553198 14234 raft_consensus.cc:374] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:10.553304 14234 raft_consensus.cc:729] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e458f45881bc4b75b8d9b775c605669f, State: Initialized, Role: FOLLOWER
I20260619 18:42:10.553325 14235 raft_consensus.cc:729] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f874daa72d6645d69024872c9ac4fe06, State: Initialized, Role: FOLLOWER
I20260619 18:42:10.554095 14235 consensus_queue.cc:260] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.554095 14234 consensus_queue.cc:260] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.555815 14214 heartbeater.cc:503] Master 127.3.85.190:38197 was elected leader, sending a full tablet report...
I20260619 18:42:10.556200 14234 ts_tablet_manager.cc:1428] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 18:42:10.556361 14235 ts_tablet_manager.cc:1428] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 18:42:10.568490 14241 raft_consensus.cc:482] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:42:10.568825 14241 raft_consensus.cc:504] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.570633 14241 leader_election.cc:288] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6be7d8d8a3b40c8ac6748fde887ac88 (127.3.85.130:41689), f874daa72d6645d69024872c9ac4fe06 (127.3.85.131:34045)
W20260619 18:42:10.575318 13947 tablet.cc:2369] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:10.578660 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.571134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:39653 (local address 127.3.85.130:41689)
0619 18:42:10.571326 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.571330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.571983 (+ 653us) server_negotiation.cc:407] Connection header received
0619 18:42:10.572252 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.572259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.572347 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.572427 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:10.573107 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.574146 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.576580 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.577574 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.577711 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.577766 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.578214 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.578370 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:42:10.578784 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.571143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:41689 (local address 127.3.85.129:39653)
0619 18:42:10.571864 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.571881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.572066 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.572481 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.572488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.572520 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:10.572928 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.572939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.574299 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.574307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.576391 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.576412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.577803 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.577810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.577986 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.578028 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.578144 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.578315 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":87,"thread_start_us":984,"threads_started":1}
I20260619 18:42:10.579183 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.571076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:52915 (local address 127.3.85.131:34045)
0619 18:42:10.571273 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.571279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.572556 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 18:42:10.572704 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.572710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.572793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.572891 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:42:10.573411 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.574426 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.577102 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.578099 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.578250 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.578318 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.578745 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.578923 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:10.579234 14242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.571143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:34045 (local address 127.3.85.129:52915)
0619 18:42:10.572436 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:10.572453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:10.572561 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:10.572928 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:10.572933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:10.572956 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:42:10.573260 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.573268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.574611 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.574616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.576921 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:10.576941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.578346 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.578352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:10.578567 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.578597 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:42:10.578704 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:42:10.578863 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":260,"spinlock_wait_cycles":40704,"thread_start_us":401,"threads_started":1}
I20260619 18:42:10.580000 14035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a943b7b35d4554932f259757689832" candidate_uuid: "e458f45881bc4b75b8d9b775c605669f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" is_pre_election: true
I20260619 18:42:10.580260 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a943b7b35d4554932f259757689832" candidate_uuid: "e458f45881bc4b75b8d9b775c605669f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f874daa72d6645d69024872c9ac4fe06" is_pre_election: true
I20260619 18:42:10.580507 14035 raft_consensus.cc:2454] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e458f45881bc4b75b8d9b775c605669f in term 0.
I20260619 18:42:10.580765 14169 raft_consensus.cc:2454] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e458f45881bc4b75b8d9b775c605669f in term 0.
I20260619 18:42:10.581266 13837 leader_election.cc:302] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6be7d8d8a3b40c8ac6748fde887ac88, e458f45881bc4b75b8d9b775c605669f; no voters:
I20260619 18:42:10.581694 14241 raft_consensus.cc:2781] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:42:10.581846 14241 raft_consensus.cc:482] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:42:10.581959 14241 raft_consensus.cc:3037] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:10.584230 14241 raft_consensus.cc:504] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.584944 14241 leader_election.cc:288] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [CANDIDATE]: Term 1 election: Requested vote from peers b6be7d8d8a3b40c8ac6748fde887ac88 (127.3.85.130:41689), f874daa72d6645d69024872c9ac4fe06 (127.3.85.131:34045)
I20260619 18:42:10.585538 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a943b7b35d4554932f259757689832" candidate_uuid: "e458f45881bc4b75b8d9b775c605669f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f874daa72d6645d69024872c9ac4fe06"
I20260619 18:42:10.585798 14169 raft_consensus.cc:3037] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:10.586067 14035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a943b7b35d4554932f259757689832" candidate_uuid: "e458f45881bc4b75b8d9b775c605669f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88"
I20260619 18:42:10.586352 14035 raft_consensus.cc:3037] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:10.588195 14169 raft_consensus.cc:2454] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e458f45881bc4b75b8d9b775c605669f in term 1.
I20260619 18:42:10.588817 13834 leader_election.cc:302] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e458f45881bc4b75b8d9b775c605669f, f874daa72d6645d69024872c9ac4fe06; no voters:
I20260619 18:42:10.588836 14035 raft_consensus.cc:2454] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e458f45881bc4b75b8d9b775c605669f in term 1.
I20260619 18:42:10.589154 14241 raft_consensus.cc:2781] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:10.589653 14241 raft_consensus.cc:686] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 1 LEADER]: Becoming Leader. State: Replica: e458f45881bc4b75b8d9b775c605669f, State: Running, Role: LEADER
I20260619 18:42:10.590159 14241 consensus_queue.cc:237] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } }
I20260619 18:42:10.595604 13757 catalog_manager.cc:5495] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f reported cstate change: term changed from 0 to 1, leader changed from <none> to e458f45881bc4b75b8d9b775c605669f (127.3.85.129). New cstate: current_term: 1 leader_uuid: "e458f45881bc4b75b8d9b775c605669f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e458f45881bc4b75b8d9b775c605669f" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260619 18:42:10.624185 14081 tablet.cc:2369] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:10.633225 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.627515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.3.85.129:38711)
0619 18:42:10.627719 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.627725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.627745 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:42:10.628103 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.628108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.628226 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.628489 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:10.628916 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.629751 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.631101 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.631509 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.631621 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.632099 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:10.632670 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:10.632784 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.632949 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.633023 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:42:10.642155 14169 raft_consensus.cc:1261] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 1 FOLLOWER]: Refusing update from remote peer e458f45881bc4b75b8d9b775c605669f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:10.642158 14035 raft_consensus.cc:1261] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 1 FOLLOWER]: Refusing update from remote peer e458f45881bc4b75b8d9b775c605669f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:10.643402 14244 consensus_queue.cc:1035] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6be7d8d8a3b40c8ac6748fde887ac88" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:10.643842 14241 consensus_queue.cc:1035] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f874daa72d6645d69024872c9ac4fe06" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:10.655392 14247 mvcc.cc:205] Tried to move back new op lower bound from 7298639997498990592 to 7298639997301727232. Current Snapshot: MvccSnapshot[applied={T|T < 7298639997498990592}]
W20260619 18:42:10.670351 14215 tablet.cc:2369] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:10.671999 13757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:10.672386 13757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:42:10.676254 13757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28fb37e2206147a2a0cf6130d60507be: Sending DeleteTablet for 3 replicas of tablet a5a943b7b35d4554932f259757689832
I20260619 18:42:10.677428 14149 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a943b7b35d4554932f259757689832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:10 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:50328
I20260619 18:42:10.677572 14015 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a943b7b35d4554932f259757689832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:10 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:50888
I20260619 18:42:10.678490 14257 tablet_replica.cc:330] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: stopping tablet replica
I20260619 18:42:10.679088 13881 tablet_service.cc:1514] Processing DeleteTablet for tablet a5a943b7b35d4554932f259757689832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:42:10 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:46790
I20260619 18:42:10.680047 14259 tablet_replica.cc:330] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: stopping tablet replica
I20260619 18:42:10.680531 14259 raft_consensus.cc:2229] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:42:10.681113 14259 raft_consensus.cc:2258] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:10.681694 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13818
I20260619 18:42:10.688843 14259 ts_tablet_manager.cc:1905] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:10.690426 14258 tablet_replica.cc:330] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: stopping tablet replica
I20260619 18:42:10.695173 14257 raft_consensus.cc:2229] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:10.696195 14257 raft_consensus.cc:2258] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:10.698736 14258 raft_consensus.cc:2229] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:42:10.699832 14258 raft_consensus.cc:2258] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:42:10.704479 14257 ts_tablet_manager.cc:1905] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:10.711278 14259 ts_tablet_manager.cc:1918] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:10.711498 14259 log.cc:1196] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/a5a943b7b35d4554932f259757689832
I20260619 18:42:10.712095 14259 ts_tablet_manager.cc:1939] T a5a943b7b35d4554932f259757689832 P e458f45881bc4b75b8d9b775c605669f: Deleting consensus metadata
I20260619 18:42:10.713982 14258 ts_tablet_manager.cc:1905] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:42:10.714068 13741 catalog_manager.cc:4841] TS e458f45881bc4b75b8d9b775c605669f (127.3.85.129:38711): tablet a5a943b7b35d4554932f259757689832 (table test-table [id=79893394b7c14e99a091ce5f593019f9]) successfully deleted
I20260619 18:42:10.714994 14257 ts_tablet_manager.cc:1918] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:10.715309 14257 log.cc:1196] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/a5a943b7b35d4554932f259757689832
I20260619 18:42:10.715965 14257 ts_tablet_manager.cc:1939] T a5a943b7b35d4554932f259757689832 P f874daa72d6645d69024872c9ac4fe06: Deleting consensus metadata
Jun 19 18:42:10 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:42:10.717937 13741 catalog_manager.cc:4841] TS f874daa72d6645d69024872c9ac4fe06 (127.3.85.131:34045): tablet a5a943b7b35d4554932f259757689832 (table test-table [id=79893394b7c14e99a091ce5f593019f9]) successfully deleted
I20260619 18:42:10.721056 14258 ts_tablet_manager.cc:1918] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:42:10.721256 14258 log.cc:1196] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/a5a943b7b35d4554932f259757689832
I20260619 18:42:10.721956 14258 ts_tablet_manager.cc:1939] T a5a943b7b35d4554932f259757689832 P b6be7d8d8a3b40c8ac6748fde887ac88: Deleting consensus metadata
I20260619 18:42:10.724046 13744 catalog_manager.cc:4841] TS b6be7d8d8a3b40c8ac6748fde887ac88 (127.3.85.130:41689): tablet a5a943b7b35d4554932f259757689832 (table test-table [id=79893394b7c14e99a091ce5f593019f9]) successfully deleted
I20260619 18:42:10.726820 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.682886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.3.85.129:38711)
0619 18:42:10.683136 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.683147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.703842 (+ 20695us) server_negotiation.cc:407] Connection header received
0619 18:42:10.704067 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.704073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.704152 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.704251 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:10.708427 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.709598 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.711648 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.712341 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.712454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.719119 (+ 6665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:10.719138 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:10.719162 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:10.719217 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:10.723620 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.724590 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.724598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.724600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.724755 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.725177 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.725186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.725188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.725564 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:10.725877 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.726189 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.726482 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:42:10.727620 13941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:10.843166 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13818
I20260619 18:42:10.862031 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13952
Jun 19 18:42:10 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:42:10.880069 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.862904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.3.85.130:41689)
0619 18:42:10.863059 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.863063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.863082 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:42:10.863254 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.863259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.863339 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.863437 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:10.864642 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.865398 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.866680 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.867066 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.867181 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:10.871900 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:10.871915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:10.871931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:10.871981 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:10.876546 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.877433 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.877443 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.877447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.877610 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:10.878062 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:10.878066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:10.878070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:10.878862 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:10.879170 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:10.879463 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 18:42:10.879697 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:10.880927 14075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:10.973289 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13952
I20260619 18:42:10.991142 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 14086
Jun 19 18:42:10 dist-test-slave-09fp krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:42:11.009290 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:10.992068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.3.85.131:34045)
0619 18:42:10.992244 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:10.992256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:10.992276 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:42:10.992443 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:10.992447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:10.992525 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:10.992613 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:10.994128 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.995270 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.996665 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:10.997103 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:10.997220 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:11.002125 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:11.002142 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:11.002159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:11.002211 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:11.005386 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:11.006313 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:11.006325 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:11.006328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:11.006989 (+ 661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:11.007480 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:11.007484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:11.007487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:11.007943 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:11.008174 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:11.008621 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 18:42:11.008905 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:42:11.010070 14209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:11.099661 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 14086
I20260619 18:42:11.117179 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 13725
I20260619 18:42:11.118333 13787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:42:11.225927 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 13725
2026-06-19T18:42:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4409 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:11 dist-test-slave-09fp krb5kdc[14267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:11 dist-test-slave-09fp krb5kdc[14267](info): set up 2 sockets
Jun 19 18:42:11 dist-test-slave-09fp krb5kdc[14267](info): commencing operation
krb5kdc: starting...
W20260619 18:42:12.355533 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:42:12 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangeradmin_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangeradmin_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangerlookup_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangerlookup_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
I20260619 18:42:12.705578 3414 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiUM6J/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/postgres -l logfile start
2026-06-19 18:42:15.466 UTC [14297] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:42:15.466 UTC [14297] LOG: listening on IPv4 address "127.3.85.148", port 54671
2026-06-19 18:42:15.470 UTC [14297] LOG: listening on Unix socket "/tmp/.s.PGSQL.54671"
2026-06-19 18:42:15.492 UTC [14300] LOG: database system was shut down at 2026-06-19 18:42:13 UTC
2026-06-19 18:42:15.496 UTC [14297] LOG: database system is ready to accept connections
I20260619 18:42:16.495244 3414 mini_postgres.cc:96] Postgres bound to 54671
2026-06-19 18:42:16.514 UTC [14308] FATAL: database "slave" does not exist
127.3.85.148:54671 - accepting connections
I20260619 18:42:16.515030 3414 mini_ranger.cc:161] Starting Ranger...
I20260619 18:42:16.543133 3414 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:42:17.116144 3414 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:42:17.116297 3414 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin
2026-06-19 18:42:17,414 [I] DB FLAVOR :POSTGRES
2026-06-19 18:42:17,414 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:42:17,414 [I] Checking connection..
2026-06-19 18:42:17,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:42:17,678 [I] Checking connection passed.
2026-06-19 18:42:17,678 [I] --------- Verifying version history table ---------
2026-06-19 18:42:17,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:42:17,942 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:42:17,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:42:18,220 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:42:18,220 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:42:18,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:42:18.464 UTC [14372] WARNING: there is no transaction in progress
2026-06-19 18:42:18,471 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:42:18,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:42:18,737 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:42:18,737 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:42:18,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:42:18,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:42:19,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-09fp.c.gcp-upstream.internal','N') ;"
2026-06-19 18:42:19,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:42:19,734 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:42:19,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:42:19,981 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:42:19,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:42:20,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:42:20,517 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:42:20,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:42:21.633 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.651 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.669 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.686 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.705 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.725 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.786 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.794 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.809 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.821 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.832 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.848 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.859 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.865 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.873 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:21.880 UTC [14480] WARNING: there is no transaction in progress
2026-06-19 18:42:22,622 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:42:22,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:42:22,891 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:42:22,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:42:23,166 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:42:23,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:42:23,452 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:42:23,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:42:23,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:42:24,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-09fp.c.gcp-upstream.internal';"
2026-06-19 18:42:24,271 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:42:24,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:42:24,511 [I] DB_PATCHES have already been applied
I20260619 18:42:24.517436 3414 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiUM6J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:42:24.517529 3414 mini_ranger.cc:193] Using host: 127.3.85.148
I20260619 18:42:24.533339 3414 mini_ranger.cc:227] Ranger admin URL: http://127.3.85.148:36311
Jun 19, 2026 6:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:42:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:42:25.695487 3414 mini_ranger.cc:160] Time spent starting Ranger: real 9.180s user 0.001s sys 0.043s
Jun 19, 2026 6:42:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:42:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 6:42:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:42:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:42:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:42:47.624285 3414 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangerkms_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/rangerkms_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.85.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.85.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:42:47 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:42:47.815551 3414 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:42:47.838203 3414 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:42:48.843195 3414 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:42:48.843372 3414 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms
I20260619 18:42:48.843502 3414 mini_ranger_kms.cc:227] Using postgres at 127.3.85.148:54671
2026-06-19 18:42:49,308 [I] DB FLAVOR :POSTGRES
2026-06-19 18:42:49,308 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:42:49,308 [I] Checking connection
2026-06-19 18:42:49,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:42:49,577 [I] connection success
2026-06-19 18:42:49,578 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:42:49,578 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:42:49,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:42:49,883 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:42:49,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:42:49,884 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:42:49,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:54671/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:42:50,202 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:42:50.763989 3414 mini_ranger_kms.cc:310] Created kms service
I20260619 18:42:50.985260 3414 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:42:51.044631 3414 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:42:51 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:42:51.540692 3414 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:42:51.541049 3414 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:42:51.541114 3414 mini_ranger_kms.cc:240] Using host: 127.3.85.148
I20260619 18:42:51.567580 3414 mini_ranger_kms.cc:276] Ranger KMS PID: 14661
I20260619 18:42:51.567787 3414 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.85.148:35385
Jun 19, 2026 6:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:42:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:42:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:42:53.951520 3414 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.136s user 0.133s sys 0.359s
I20260619 18:42:53.951833 3414 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:42:53.952065 3414 mini_ranger_kms.cc:401] 127.3.85.148:35385/kms/v1/keys
Jun 19 18:42:54 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:42:54 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
Jun 19 18:42:54 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
Jun 19 18:42:57 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:42:57.342624 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34667
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:34667
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:35385/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:57.716733 14711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:57.717130 14711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:57.717290 14711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:57.725036 14711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:57.725200 14711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:57.725299 14711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:57.725410 14711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:57.725497 14711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:57.736730 14711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.85.148:35385/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:34667
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:57.738240 14711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:57.739750 14711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:57.753131 14716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:57.753670 14717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:57.753886 14719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:57.754145 14711 server_base.cc:1034] running on GCE node
I20260619 18:42:57.755029 14711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:57.756624 14711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:57.757862 14711 hybrid_clock.cc:648] HybridClock initialized: now 1781894577757808 us; error 54 us; skew 500 ppm
I20260619 18:42:57.758227 14711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:57 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:57.766350 14711 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:42:57.768196 14711 webserver.cc:460] Webserver started at http://127.3.85.190:36803/ using document root <none> and password file <none>
I20260619 18:42:57.768949 14711 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:57.769095 14711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:57.769424 14711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:42:57 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:42:57.833518 14711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "dbd28943cf4b482999ce0a02cf9b0db2"
format_stamp: "Formatted at 2026-06-19 18:42:57 on dist-test-slave-09fp"
server_key: "86970edcf3fc22b55811ef2b28340b7b"
server_key_iv: "17ebcfc11fc1de0391b0876920980eb7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:57.834578 14711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "dbd28943cf4b482999ce0a02cf9b0db2"
format_stamp: "Formatted at 2026-06-19 18:42:57 on dist-test-slave-09fp"
server_key: "86970edcf3fc22b55811ef2b28340b7b"
server_key_iv: "17ebcfc11fc1de0391b0876920980eb7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:57.840493 14711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:42:57.861673 14726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:57.863411 14711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:42:57.863636 14711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "dbd28943cf4b482999ce0a02cf9b0db2"
format_stamp: "Formatted at 2026-06-19 18:42:57 on dist-test-slave-09fp"
server_key: "86970edcf3fc22b55811ef2b28340b7b"
server_key_iv: "17ebcfc11fc1de0391b0876920980eb7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:57.863862 14711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:57.878337 14711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:57.882128 14711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:57.882529 14711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:57.903644 14711 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:34667
I20260619 18:42:57.903666 14777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:34667 every 8 connection(s)
I20260619 18:42:57.905200 14711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:57.909294 14778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:57.915457 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 14711
I20260619 18:42:57.915675 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:42:57.916462 3414 external_mini_cluster.cc:1428] Setting key acbd24f6d9d6089f723bc501021e2151
I20260619 18:42:57.921770 14778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: Bootstrap starting.
I20260619 18:42:57.925701 14778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:57.927134 14778 log.cc:824] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:57.931077 14778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: No bootstrap required, opened a new log
Jun 19 18:42:57 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:42:57.940104 14778 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } }
I20260619 18:42:57.940549 14778 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:57.940672 14778 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd28943cf4b482999ce0a02cf9b0db2, State: Initialized, Role: FOLLOWER
I20260619 18:42:57.941375 14778 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } }
I20260619 18:42:57.941666 14778 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:57.941836 14778 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:57.942034 14778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:57.942647 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:57.918820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.3.85.190:34667)
0619 18:42:57.920159 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:57.920172 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:57.920213 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:42:57.921932 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:57.921976 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:57.922386 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:57.922793 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:57.924491 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:57.925900 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:57.927772 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:57.928938 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:57.929093 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:57.933975 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:57.933996 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:57.934021 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:57.934076 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:57.937833 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:57.938803 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:57.938811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:57.938818 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:57.938997 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:57.939414 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:57.939419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:57.939421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:57.940040 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:57.940384 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:57.940825 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 18:42:57.941295 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":874,"threads_started":1}
I20260619 18:42:57.944609 14778 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } }
I20260619 18:42:57.945235 14778 leader_election.cc:302] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd28943cf4b482999ce0a02cf9b0db2; no voters:
I20260619 18:42:57.945977 14778 leader_election.cc:288] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:57.946914 14783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:57.948185 14783 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [term 1 LEADER]: Becoming Leader. State: Replica: dbd28943cf4b482999ce0a02cf9b0db2, State: Running, Role: LEADER
I20260619 18:42:57.948558 14778 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:57.948808 14783 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } }
I20260619 18:42:57.951898 14785 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd28943cf4b482999ce0a02cf9b0db2. Latest consensus state: current_term: 1 leader_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } } }
I20260619 18:42:57.952327 14785 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:57.952457 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd28943cf4b482999ce0a02cf9b0db2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34667 } } }
I20260619 18:42:57.952801 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:57.955650 14792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:57.962224 14792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:57.963057 14778 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:42:57.984968 14792 catalog_manager.cc:1347] Generated new cluster ID: 63e029dbbd064a0ab2786b6ff68fd60f
I20260619 18:42:57.985248 14792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:58.011029 14792 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:58.013607 14792 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:58.031693 14792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: Generated new TSK 0
I20260619 18:42:58.033083 14792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:42:58.768553 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:35385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:42:59.249642 14823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:59.250072 14823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:59.250295 14823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:59.262477 14823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:59.262653 14823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:59.262882 14823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:42:59.279865 14823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.85.148:35385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:42:59.281689 14823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:59.283605 14823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:59.301642 14833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:59.304167 14834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:59.315783 14823 server_base.cc:1034] running on GCE node
W20260619 18:42:59.317250 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:59.318142 14823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:59.319360 14823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:59.320617 14823 hybrid_clock.cc:648] HybridClock initialized: now 1781894579320554 us; error 63 us; skew 500 ppm
I20260619 18:42:59.320990 14823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:59.324568 14823 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:42:59.326402 14823 webserver.cc:460] Webserver started at http://127.3.85.129:45671/ using document root <none> and password file <none>
I20260619 18:42:59.327618 14823 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:59.327780 14823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:59.328132 14823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:42:59.349210 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "543cfcf86d82435883a9382a59a606a3"
format_stamp: "Formatted at 2026-06-19 18:42:59 on dist-test-slave-09fp"
server_key: "800cbe5857d2190b1a9a8dcb5a214f69"
server_key_iv: "4f96aab8eeae7c310de9e66f5b7dbff7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:59.350529 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "543cfcf86d82435883a9382a59a606a3"
format_stamp: "Formatted at 2026-06-19 18:42:59 on dist-test-slave-09fp"
server_key: "800cbe5857d2190b1a9a8dcb5a214f69"
server_key_iv: "4f96aab8eeae7c310de9e66f5b7dbff7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:59.364878 14823 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260619 18:42:59.387149 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:59.389356 14823 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260619 18:42:59.389581 14823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "543cfcf86d82435883a9382a59a606a3"
format_stamp: "Formatted at 2026-06-19 18:42:59 on dist-test-slave-09fp"
server_key: "800cbe5857d2190b1a9a8dcb5a214f69"
server_key_iv: "4f96aab8eeae7c310de9e66f5b7dbff7"
server_key_version: "kuduclusterkey@0"
I20260619 18:42:59.389945 14823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:59.435616 14823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:59.441373 14823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:59.441825 14823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:59.454731 14823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:42:59.463124 14823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:42:59.463320 14823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 18:42:59.463620 14823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:42:59.463747 14823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:42:59.674934 14823 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:33431
I20260619 18:42:59.674944 14955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:33431 every 8 connection(s)
I20260619 18:42:59.676474 14823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:42:59.678726 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 14823
I20260619 18:42:59.678941 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:42:59.679776 3414 external_mini_cluster.cc:1428] Setting key aa2694727df8332130b0a7e1700b6543
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:42:59.720158 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.683865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:43301 (local address 127.3.85.190:34667)
0619 18:42:59.684648 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:59.684654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:59.692417 (+ 7763us) server_negotiation.cc:407] Connection header received
0619 18:42:59.697512 (+ 5095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:59.697523 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:59.697651 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:59.697737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:59.700501 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.701760 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.704874 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.705654 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.705799 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.711896 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:59.711907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:59.711914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:59.711968 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:59.715617 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:59.716677 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:59.716686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:59.716694 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:59.716863 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:59.717532 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:59.717540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:59.717546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:59.717987 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:59.718135 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:59.719342 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0619 18:42:59.719645 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":785,"threads_started":1}
I20260619 18:42:59.723220 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:59.681180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34667 (local address 127.3.85.129:43301)
0619 18:42:59.692172 (+ 10992us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:59.692255 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:59.697078 (+ 4823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:59.698402 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:59.698418 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:59.699185 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:59.700184 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.700219 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.702106 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.702115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.704612 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:59.704650 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:59.706315 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:59.706326 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:59.706740 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:59.707693 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:59.707733 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:59.711507 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:59.715877 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:59.715886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:59.715904 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:59.716506 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:59.717054 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:59.717061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:59.717064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:59.717390 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:59.718244 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:59.718257 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:59.718862 (+ 605us) client_negotiation.cc:769] Sending connection context
0619 18:42:59.720838 (+ 1976us) client_negotiation.cc:240] Negotiation successful
0619 18:42:59.721523 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10554,"thread_start_us":11093,"threads_started":1}
I20260619 18:42:59.725699 14956 heartbeater.cc:347] Connected to a master server at 127.3.85.190:34667
I20260619 18:42:59.726078 14956 heartbeater.cc:464] Registering TS with master...
I20260619 18:42:59.727113 14956 heartbeater.cc:511] Master 127.3.85.190:34667 requested a full tablet report, sending...
I20260619 18:42:59.730238 14743 ts_manager.cc:194] Registered new tserver with Master: 543cfcf86d82435883a9382a59a606a3 (127.3.85.129:33431)
I20260619 18:42:59.732724 14743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:43301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:42:59.754640 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:35385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:42:59 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894579, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
W20260619 18:43:00.263360 14962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:00.263731 14962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:00.263901 14962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:00.272007 14962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:00.272179 14962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:00.272368 14962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:43:00.283011 14962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.85.148:35385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:00.284438 14962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:00.285794 14962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:00.300787 14977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.300889 14976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.300863 14979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:00.302299 14962 server_base.cc:1034] running on GCE node
I20260619 18:43:00.302997 14962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:00.304069 14962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:00.305297 14962 hybrid_clock.cc:648] HybridClock initialized: now 1781894580305248 us; error 36 us; skew 500 ppm
I20260619 18:43:00.305676 14962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:00 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:00.309029 14962 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:43:00.310657 14962 webserver.cc:460] Webserver started at http://127.3.85.130:45579/ using document root <none> and password file <none>
I20260619 18:43:00.311389 14962 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:00.311540 14962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:00.311875 14962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:00 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:00.326198 14962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "2806eee5b22a4343a1403a4a6f759a4a"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "0c673306f237fc014ada48b80e6d4580"
server_key_iv: "56f01981d43fd51740ad49f161b28591"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.327270 14962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "2806eee5b22a4343a1403a4a6f759a4a"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "0c673306f237fc014ada48b80e6d4580"
server_key_iv: "56f01981d43fd51740ad49f161b28591"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.333513 14962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 18:43:00.347796 14986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.349671 14962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:43:00.349927 14962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "2806eee5b22a4343a1403a4a6f759a4a"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "0c673306f237fc014ada48b80e6d4580"
server_key_iv: "56f01981d43fd51740ad49f161b28591"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.350144 14962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:00.390512 14962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:00.395103 14962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:00.395385 14962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:00.396415 14962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:00.397878 14962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:00.397956 14962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.398044 14962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:00.398089 14962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.434965 14962 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:33425
I20260619 18:43:00.435045 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:33425 every 8 connection(s)
I20260619 18:43:00.436378 14962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:00.443287 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 14962
I20260619 18:43:00.443496 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:00.443928 3414 external_mini_cluster.cc:1428] Setting key 264d192cd81dd62b60f0629224476faa
Jun 19 18:43:00 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:00.459194 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.439581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45683 (local address 127.3.85.190:34667)
0619 18:43:00.440105 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:00.440110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:00.441062 (+ 952us) server_negotiation.cc:407] Connection header received
0619 18:43:00.442992 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:00.442998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:00.443107 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:00.443227 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:00.444898 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.445701 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.447511 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.447937 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.448089 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.451880 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:00.451886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:00.451889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:00.451925 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:00.455401 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.456486 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.456491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.456493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.456649 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:00.457229 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:00.457233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:00.457235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:00.457537 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:00.457649 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:00.458711 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0619 18:43:00.458909 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":308,"threads_started":1}
I20260619 18:43:00.460791 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:00.440107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34667 (local address 127.3.85.130:45683)
0619 18:43:00.440877 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:00.440918 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:00.442686 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:00.443531 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:00.443542 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:00.444058 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:00.444712 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.444731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.445881 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.445885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.447325 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:00.447341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:00.448198 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:00.448206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:00.448463 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:00.449217 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:00.449245 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:00.451610 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:00.455569 (+ 3959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.455582 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.455603 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.456354 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.456781 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:00.456788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:00.456795 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:00.457099 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:00.457671 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:00.457683 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:00.458197 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 18:43:00.458558 (+ 361us) client_negotiation.cc:240] Negotiation successful
0619 18:43:00.459177 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":345,"threads_started":1}
I20260619 18:43:00.462469 15099 heartbeater.cc:347] Connected to a master server at 127.3.85.190:34667
I20260619 18:43:00.462831 15099 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:00.463779 15099 heartbeater.cc:511] Master 127.3.85.190:34667 requested a full tablet report, sending...
I20260619 18:43:00.465929 14743 ts_manager.cc:194] Registered new tserver with Master: 2806eee5b22a4343a1403a4a6f759a4a (127.3.85.130:33425)
I20260619 18:43:00.466814 14743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:45683
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:43:00.498701 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:35385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:43:00.736588 14956 heartbeater.cc:503] Master 127.3.85.190:34667 was elected leader, sending a full tablet report...
W20260619 18:43:00.864815 15105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:00.865152 15105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:00.865346 15105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:00.873472 15105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:00.873628 15105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:00.873835 15105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:43:00.885012 15105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.85.148:35385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:34667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:00.886478 15105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:00.887902 15105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:00.902441 15111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:00.903195 15113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:00.903762 15105 server_base.cc:1034] running on GCE node
W20260619 18:43:00.906590 15110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:00.907445 15105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:00.908537 15105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:00.909795 15105 hybrid_clock.cc:648] HybridClock initialized: now 1781894580909740 us; error 59 us; skew 500 ppm
I20260619 18:43:00.910153 15105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:00 dist-test-slave-09fp krb5kdc[14267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:00.913548 15105 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:43:00.915266 15105 webserver.cc:460] Webserver started at http://127.3.85.131:39145/ using document root <none> and password file <none>
I20260619 18:43:00.916020 15105 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:00.916172 15105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:00.916502 15105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:00 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:00.931895 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "72bd147c5f994d85a90a2587e27ff039"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "15d6dc9310e2401b564d224a27e1e60d"
server_key_iv: "8584c9f7183ab5b77ffe05701b7ac31e"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.932890 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "72bd147c5f994d85a90a2587e27ff039"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "15d6dc9310e2401b564d224a27e1e60d"
server_key_iv: "8584c9f7183ab5b77ffe05701b7ac31e"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.939846 15105 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:43:00.957165 15120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.958943 15105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:43:00.959142 15105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "72bd147c5f994d85a90a2587e27ff039"
format_stamp: "Formatted at 2026-06-19 18:43:00 on dist-test-slave-09fp"
server_key: "15d6dc9310e2401b564d224a27e1e60d"
server_key_iv: "8584c9f7183ab5b77ffe05701b7ac31e"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:00.959393 15105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:00.977726 15105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:00.982049 15105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:00.982390 15105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:00.983520 15105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:00.984993 15105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:00.985107 15105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:00.985275 15105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:00.985352 15105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:01.023382 15105 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46425
I20260619 18:43:01.023416 15232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46425 every 8 connection(s)
I20260619 18:43:01.024817 15105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:01.034133 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 15105
I20260619 18:43:01.034368 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:01.035079 3414 external_mini_cluster.cc:1428] Setting key 3ffcf6b93ac86a317c6708600dcbcc27
Jun 19 18:43:01 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894580, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:01.055352 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.027931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:46271 (local address 127.3.85.190:34667)
0619 18:43:01.028456 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.028464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.029584 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 18:43:01.032046 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.032052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.032147 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.032267 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.034183 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.035642 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.039040 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.039710 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.039865 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.046036 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.046043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.046046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.046084 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.051063 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.052021 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.052028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.052032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.052223 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.052859 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.052865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.052870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.053301 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.053476 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.054500 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.054785 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260619 18:43:01.056383 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.028555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34667 (local address 127.3.85.131:46271)
0619 18:43:01.029388 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.029436 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.031678 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.032562 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.032572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.033064 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:01.033876 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.033911 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.035908 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.035915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.038596 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.038631 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.040096 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.040326 (+ 230us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.040706 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.041786 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:01.041822 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:01.045480 (+ 3658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:01.051270 (+ 5790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.051282 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.051304 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.051823 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.052380 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:01.052385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:01.052388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:01.052685 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:01.053471 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:01.053480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:01.053998 (+ 518us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.054343 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.054993 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260619 18:43:01.057946 15233 heartbeater.cc:347] Connected to a master server at 127.3.85.190:34667
I20260619 18:43:01.058323 15233 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:01.059286 15233 heartbeater.cc:511] Master 127.3.85.190:34667 requested a full tablet report, sending...
I20260619 18:43:01.061060 14743 ts_manager.cc:194] Registered new tserver with Master: 72bd147c5f994d85a90a2587e27ff039 (127.3.85.131:46425)
I20260619 18:43:01.062052 14743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:46271
I20260619 18:43:01.068676 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:43:01.096276 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.079802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.3.85.190:34667)
0619 18:43:01.080013 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.080021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.080609 (+ 588us) server_negotiation.cc:407] Connection header received
0619 18:43:01.080913 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.080922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.081042 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.081167 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.082977 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.084209 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.085743 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.086332 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.086487 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.089919 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.089925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.089928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.089965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.093065 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.094001 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.094007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.094011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.094164 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.094665 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.094671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.094674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.094974 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.095250 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.095603 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.095928 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:43:01.101004 14743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:01.109313 14743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:01.149713 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.124396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.3.85.131:46425)
0619 18:43:01.139129 (+ 14733us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.139136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.139170 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:43:01.139268 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.139274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.139513 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.139781 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.141618 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.143040 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.146762 (+ 3722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.148377 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.148527 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.148789 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.149121 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.149315 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14546,"thread_start_us":14392,"threads_started":1}
I20260619 18:43:01.149744 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.124298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46425 (local address 127.0.0.1:57938)
0619 18:43:01.125166 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.125211 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.125407 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.139805 (+ 14398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.139814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.139858 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.141359 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.141381 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.143203 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.143212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.146082 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.146097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.148615 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.148623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.148843 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.148897 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.149051 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.149260 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":411,"threads_started":1}
I20260619 18:43:01.152020 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.124863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45096)
0619 18:43:01.136289 (+ 11426us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.136314 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.136469 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.143508 (+ 7039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.143513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.143545 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.144086 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.144100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.146115 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.146120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.148769 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.148786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.150881 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.150889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.151058 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.151099 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.151455 (+ 356us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.151622 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11266,"mutex_wait_us":661,"thread_start_us":10464,"threads_started":1}
I20260619 18:43:01.154083 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.125301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.3.85.129:33431)
0619 18:43:01.142209 (+ 16908us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.142222 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.142266 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:43:01.142406 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.142419 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.143275 (+ 856us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.143516 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.144307 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.145937 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.149040 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.150659 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.151147 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.151418 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.151503 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.151688 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16630,"thread_start_us":17585,"threads_started":1}
I20260619 18:43:01.154151 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.125794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:33425 (local address 127.0.0.1:45054)
0619 18:43:01.144251 (+ 18457us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.144272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.144394 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.144912 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.144917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.144942 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.145323 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.145332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.147168 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.147172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.149555 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.149572 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.153183 (+ 3611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.153192 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.153390 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.153423 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.153656 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.153771 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18284,"mutex_wait_us":2,"thread_start_us":18105,"threads_started":1}
I20260619 18:43:01.155301 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.126051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.3.85.130:33425)
0619 18:43:01.142878 (+ 16827us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.142887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.144364 (+ 1477us) server_negotiation.cc:407] Connection header received
0619 18:43:01.144543 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.144549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.144722 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.144928 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.145494 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.146887 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.149994 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.152652 (+ 2658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.152810 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.153015 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.154760 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.154922 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16686,"thread_start_us":16576,"threads_started":1}
I20260619 18:43:01.155113 15168 tablet_service.cc:1467] Processing CreateTablet for tablet bf636350869d4064ad782530bbb68e64 (DEFAULT_TABLE table=test-table [id=201caeb3e6594cbb9766fd561d361d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.156781 15168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf636350869d4064ad782530bbb68e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.157976 14891 tablet_service.cc:1467] Processing CreateTablet for tablet bf636350869d4064ad782530bbb68e64 (DEFAULT_TABLE table=test-table [id=201caeb3e6594cbb9766fd561d361d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.158993 15034 tablet_service.cc:1467] Processing CreateTablet for tablet bf636350869d4064ad782530bbb68e64 (DEFAULT_TABLE table=test-table [id=201caeb3e6594cbb9766fd561d361d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:01.159757 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf636350869d4064ad782530bbb68e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.160343 15034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf636350869d4064ad782530bbb68e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:01.173457 15252 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Bootstrap starting.
I20260619 18:43:01.177222 15252 tablet_bootstrap.cc:654] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.178665 15252 log.cc:824] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.179121 15253 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Bootstrap starting.
I20260619 18:43:01.184271 15253 tablet_bootstrap.cc:654] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.185817 15254 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Bootstrap starting.
I20260619 18:43:01.185860 15253 log.cc:824] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.189478 15253 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: No bootstrap required, opened a new log
I20260619 18:43:01.189797 15253 ts_tablet_manager.cc:1397] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260619 18:43:01.189863 15252 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: No bootstrap required, opened a new log
I20260619 18:43:01.190166 15252 ts_tablet_manager.cc:1397] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260619 18:43:01.191269 15254 tablet_bootstrap.cc:654] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:01.192689 15254 log.cc:824] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:01.198873 15253 raft_consensus.cc:348] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.199462 15253 raft_consensus.cc:374] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.199581 15253 raft_consensus.cc:729] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72bd147c5f994d85a90a2587e27ff039, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.200340 15253 consensus_queue.cc:260] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.202104 15254 tablet_bootstrap.cc:492] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: No bootstrap required, opened a new log
I20260619 18:43:01.202512 15254 ts_tablet_manager.cc:1397] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260619 18:43:01.203152 15233 heartbeater.cc:503] Master 127.3.85.190:34667 was elected leader, sending a full tablet report...
I20260619 18:43:01.202873 15252 raft_consensus.cc:348] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.203382 15252 raft_consensus.cc:374] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.203495 15252 raft_consensus.cc:729] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543cfcf86d82435883a9382a59a606a3, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.203838 15253 ts_tablet_manager.cc:1428] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 18:43:01.204195 15252 consensus_queue.cc:260] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.206686 15252 ts_tablet_manager.cc:1428] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Time spent starting tablet: real 0.016s user 0.014s sys 0.005s
I20260619 18:43:01.212965 15254 raft_consensus.cc:348] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.213398 15254 raft_consensus.cc:374] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:01.213531 15254 raft_consensus.cc:729] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2806eee5b22a4343a1403a4a6f759a4a, State: Initialized, Role: FOLLOWER
I20260619 18:43:01.214170 15254 consensus_queue.cc:260] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.215238 15099 heartbeater.cc:503] Master 127.3.85.190:34667 was elected leader, sending a full tablet report...
I20260619 18:43:01.215757 15254 ts_tablet_manager.cc:1428] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
W20260619 18:43:01.276698 15234 tablet.cc:2369] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:01.299649 15259 raft_consensus.cc:482] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:01.299980 15259 raft_consensus.cc:504] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.301756 15259 leader_election.cc:288] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72bd147c5f994d85a90a2587e27ff039 (127.3.85.131:46425), 2806eee5b22a4343a1403a4a6f759a4a (127.3.85.130:33425)
I20260619 18:43:01.308820 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.302146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:38487 (local address 127.3.85.131:46425)
0619 18:43:01.302349 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.302354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.303427 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 18:43:01.303690 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.303699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.303803 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.303900 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.304489 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.305251 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.307076 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.307876 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.308071 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.308157 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.308418 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.308550 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:43:01.308912 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.302139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46425 (local address 127.3.85.129:38487)
0619 18:43:01.303308 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.303325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.303515 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.304000 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.304004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.304031 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.304351 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.304359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.305376 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.305383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.306906 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.306919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.308088 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.308094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.308260 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.308284 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.308392 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.308552 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":437,"threads_started":1}
I20260619 18:43:01.309294 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.303736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:38869 (local address 127.3.85.130:33425)
0619 18:43:01.303946 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.303951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.304167 (+ 216us) server_negotiation.cc:407] Connection header received
0619 18:43:01.304392 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.304396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.304461 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.304558 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:01.305065 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.305740 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.307538 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.308323 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.308506 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.308585 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.308882 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.309049 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:43:01.309877 15188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf636350869d4064ad782530bbb68e64" candidate_uuid: "543cfcf86d82435883a9382a59a606a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bd147c5f994d85a90a2587e27ff039" is_pre_election: true
I20260619 18:43:01.310323 15188 raft_consensus.cc:2454] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543cfcf86d82435883a9382a59a606a3 in term 0.
I20260619 18:43:01.310420 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.303488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:33425 (local address 127.3.85.129:38869)
0619 18:43:01.304057 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.304070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:01.304232 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:01.304576 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:01.304580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:01.304603 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:01.304921 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.304929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.305889 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.305898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.307373 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:01.307385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.308550 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.308557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:01.308718 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.308741 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:43:01.308844 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:43:01.309307 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260619 18:43:01.311080 14846 leader_election.cc:302] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 543cfcf86d82435883a9382a59a606a3, 72bd147c5f994d85a90a2587e27ff039; no voters:
I20260619 18:43:01.311575 15259 raft_consensus.cc:2781] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:01.311527 15054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf636350869d4064ad782530bbb68e64" candidate_uuid: "543cfcf86d82435883a9382a59a606a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2806eee5b22a4343a1403a4a6f759a4a" is_pre_election: true
I20260619 18:43:01.312214 15259 raft_consensus.cc:482] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:01.312232 15054 raft_consensus.cc:2454] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543cfcf86d82435883a9382a59a606a3 in term 0.
I20260619 18:43:01.312351 15259 raft_consensus.cc:3037] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.314713 15259 raft_consensus.cc:504] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.315330 15259 leader_election.cc:288] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [CANDIDATE]: Term 1 election: Requested vote from peers 72bd147c5f994d85a90a2587e27ff039 (127.3.85.131:46425), 2806eee5b22a4343a1403a4a6f759a4a (127.3.85.130:33425)
I20260619 18:43:01.316119 15188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf636350869d4064ad782530bbb68e64" candidate_uuid: "543cfcf86d82435883a9382a59a606a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bd147c5f994d85a90a2587e27ff039"
I20260619 18:43:01.316171 15054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf636350869d4064ad782530bbb68e64" candidate_uuid: "543cfcf86d82435883a9382a59a606a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2806eee5b22a4343a1403a4a6f759a4a"
I20260619 18:43:01.316334 15188 raft_consensus.cc:3037] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.316391 15054 raft_consensus.cc:3037] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:01.318681 15188 raft_consensus.cc:2454] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543cfcf86d82435883a9382a59a606a3 in term 1.
I20260619 18:43:01.318884 15054 raft_consensus.cc:2454] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543cfcf86d82435883a9382a59a606a3 in term 1.
I20260619 18:43:01.319206 14846 leader_election.cc:302] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 543cfcf86d82435883a9382a59a606a3, 72bd147c5f994d85a90a2587e27ff039; no voters:
I20260619 18:43:01.319579 15259 raft_consensus.cc:2781] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:01.320034 15259 raft_consensus.cc:686] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 1 LEADER]: Becoming Leader. State: Replica: 543cfcf86d82435883a9382a59a606a3, State: Running, Role: LEADER
I20260619 18:43:01.320516 15259 consensus_queue.cc:237] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } }
I20260619 18:43:01.325341 14743 catalog_manager.cc:5495] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 543cfcf86d82435883a9382a59a606a3 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "543cfcf86d82435883a9382a59a606a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543cfcf86d82435883a9382a59a606a3" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33431 } health_report { overall_health: HEALTHY } } }
I20260619 18:43:01.341876 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.336710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.3.85.129:33431)
0619 18:43:01.336922 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.336927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.336945 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:43:01.337060 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.337064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.337150 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.337279 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:01.337605 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.338331 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.339774 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.340172 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.340268 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.340736 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:01.341278 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:01.341393 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.341493 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.341563 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:43:01.350818 15054 raft_consensus.cc:1261] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 1 FOLLOWER]: Refusing update from remote peer 543cfcf86d82435883a9382a59a606a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:01.350818 15188 raft_consensus.cc:1261] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 1 FOLLOWER]: Refusing update from remote peer 543cfcf86d82435883a9382a59a606a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:01.351869 15259 consensus_queue.cc:1035] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72bd147c5f994d85a90a2587e27ff039" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:01.352205 15263 consensus_queue.cc:1035] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2806eee5b22a4343a1403a4a6f759a4a" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:01.363384 15266 mvcc.cc:205] Tried to move back new op lower bound from 7298640205200207872 to 7298640205092155392. Current Snapshot: MvccSnapshot[applied={T|T < 7298640205200207872}]
I20260619 18:43:01.368167 15268 mvcc.cc:205] Tried to move back new op lower bound from 7298640205200207872 to 7298640205092155392. Current Snapshot: MvccSnapshot[applied={T|T < 7298640205200207872}]
I20260619 18:43:01.382237 14743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:01.382578 14743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:01.386834 14743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbd28943cf4b482999ce0a02cf9b0db2: Sending DeleteTablet for 3 replicas of tablet bf636350869d4064ad782530bbb68e64
I20260619 18:43:01.388144 15168 tablet_service.cc:1514] Processing DeleteTablet for tablet bf636350869d4064ad782530bbb68e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:57938
I20260619 18:43:01.388387 15034 tablet_service.cc:1514] Processing DeleteTablet for tablet bf636350869d4064ad782530bbb68e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45054
I20260619 18:43:01.388438 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet bf636350869d4064ad782530bbb68e64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:01 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:45096
I20260619 18:43:01.394562 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 14823
I20260619 18:43:01.399178 15276 tablet_replica.cc:330] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: stopping tablet replica
I20260619 18:43:01.400568 15276 raft_consensus.cc:2229] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:01.402117 15276 raft_consensus.cc:2258] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.406402 15276 ts_tablet_manager.cc:1905] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.414835 15275 tablet_replica.cc:330] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: stopping tablet replica
I20260619 18:43:01.415386 15275 raft_consensus.cc:2229] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:01.416005 15275 raft_consensus.cc:2258] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.419296 15276 ts_tablet_manager.cc:1918] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:01.419687 15276 log.cc:1196] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/bf636350869d4064ad782530bbb68e64
I20260619 18:43:01.420606 15276 ts_tablet_manager.cc:1939] T bf636350869d4064ad782530bbb68e64 P 2806eee5b22a4343a1403a4a6f759a4a: Deleting consensus metadata
Jun 19 18:43:01 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:43:01.423134 14729 catalog_manager.cc:4841] TS 2806eee5b22a4343a1403a4a6f759a4a (127.3.85.130:33425): tablet bf636350869d4064ad782530bbb68e64 (table test-table [id=201caeb3e6594cbb9766fd561d361d0a]) successfully deleted
I20260619 18:43:01.425027 15275 ts_tablet_manager.cc:1905] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.427078 15277 tablet_replica.cc:330] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: stopping tablet replica
I20260619 18:43:01.427551 15277 raft_consensus.cc:2229] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:01.428186 15277 raft_consensus.cc:2258] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:01.430778 15277 ts_tablet_manager.cc:1905] T bf636350869d4064ad782530bbb68e64 P 543cfcf86d82435883a9382a59a606a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:01.433768 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.396841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.3.85.129:33431)
0619 18:43:01.397131 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.397140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.407507 (+ 10367us) server_negotiation.cc:407] Connection header received
0619 18:43:01.408151 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.408161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.408288 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.408387 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.410431 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.412163 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.416499 (+ 4336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.417263 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.417402 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.424689 (+ 7287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.424714 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.424734 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.424796 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.429548 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.430714 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.430724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.430728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.431001 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.431524 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.431530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.431533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.432053 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.432364 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.433053 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.433349 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 18:43:01.436048 14951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:01.436432 15275 ts_tablet_manager.cc:1918] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:01.436674 15275 log.cc:1196] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/bf636350869d4064ad782530bbb68e64
I20260619 18:43:01.437333 15275 ts_tablet_manager.cc:1939] T bf636350869d4064ad782530bbb68e64 P 72bd147c5f994d85a90a2587e27ff039: Deleting consensus metadata
I20260619 18:43:01.439952 14729 catalog_manager.cc:4841] TS 72bd147c5f994d85a90a2587e27ff039 (127.3.85.131:46425): tablet bf636350869d4064ad782530bbb68e64 (table test-table [id=201caeb3e6594cbb9766fd561d361d0a]) successfully deleted
I20260619 18:43:01.537536 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 14823
I20260619 18:43:01.556886 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 14962
W20260619 18:43:01.556862 14728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:33431 (error 108)
I20260619 18:43:01.558852 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.558323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45120)
0619 18:43:01.558545 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.558660 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:43:01.559187 14728 catalog_manager.cc:4584] TS 543cfcf86d82435883a9382a59a606a3 (127.3.85.129:33431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf636350869d4064ad782530bbb68e64: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Jun 19 18:43:01 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:43:01.573283 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.557781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.3.85.130:33425)
0619 18:43:01.557972 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.557976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.558094 (+ 118us) server_negotiation.cc:407] Connection header received
0619 18:43:01.558273 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.558280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.558384 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.558513 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.560037 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.560801 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.562094 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.562576 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.562684 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.567358 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.567392 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.567417 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.567495 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.570666 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.571387 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.571394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.571397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.571543 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.571895 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.571898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.571900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.572346 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.572551 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.572837 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.573021 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:43:01.574002 15094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:01.613118 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.612611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45122)
0619 18:43:01.612848 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.612960 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:43:01.614233 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.613885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45126)
0619 18:43:01.614033 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.614119 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:43:01.665405 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 14962
I20260619 18:43:01.675094 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.674404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45138)
0619 18:43:01.674736 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.674862 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 18:43:01.676324 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.675832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45148)
0619 18:43:01.676041 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.676146 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:43:01.683185 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 15105
Jun 19 18:43:01 dist-test-slave-09fp krb5kdc[14267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:43:01.700193 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.684010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.3.85.131:46425)
0619 18:43:01.684206 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:01.684211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:01.684232 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:43:01.684437 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:01.684441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:01.684522 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:01.684632 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:01.686014 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.686846 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.688314 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:01.688744 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:01.688898 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:01.693798 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:01.693816 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:01.693833 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:01.693882 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:01.696966 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.697864 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.697873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.697876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.698019 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:01.698419 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:01.698427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:01.698433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:01.698926 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:01.699244 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:01.699594 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 18:43:01.699827 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:43:01.701025 15228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:01.757447 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.756938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45154)
0619 18:43:01.757158 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.757253 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 18:43:01.758728 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:01.758246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33431 (local address 127.0.0.1:45158)
0619 18:43:01.758520 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:01.758598 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 18:43:01.792199 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 15105
I20260619 18:43:01.809659 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 14711
I20260619 18:43:01.811028 14773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:01.925731 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 14711
I20260619 18:43:01.947887 3414 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:43:02.284663 3414 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:43:02.284883 3414 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:43:02.631639 3414 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:43:02.632 UTC [14297] LOG: received smart shutdown request
2026-06-19 18:43:02.636 UTC [14297] LOG: background worker "logical replication launcher" (PID 14306) exited with exit code 1
2026-06-19 18:43:02.636 UTC [14301] LOG: shutting down
2026-06-19 18:43:02.844 UTC [14297] LOG: database system is shut down
2026-06-19T18:43:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51739 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:43:03 dist-test-slave-09fp krb5kdc[15292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:43:03 dist-test-slave-09fp krb5kdc[15292](info): set up 2 sockets
Jun 19 18:43:03 dist-test-slave-09fp krb5kdc[15292](info): commencing operation
krb5kdc: starting...
W20260619 18:43:04.106901 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:43:04 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:43:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangeradmin_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangeradmin_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangerlookup_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangerlookup_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
I20260619 18:43:04.458365 3414 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHiUM6J/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/postgres -l logfile start
2026-06-19 18:43:07.074 UTC [15322] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:43:07.074 UTC [15322] LOG: listening on IPv4 address "127.3.85.148", port 57777
2026-06-19 18:43:07.078 UTC [15322] LOG: listening on Unix socket "/tmp/.s.PGSQL.57777"
2026-06-19 18:43:07.100 UTC [15325] LOG: database system was shut down at 2026-06-19 18:43:05 UTC
2026-06-19 18:43:07.105 UTC [15322] LOG: database system is ready to accept connections
I20260619 18:43:08.094969 3414 mini_postgres.cc:96] Postgres bound to 57777
2026-06-19 18:43:08.112 UTC [15333] FATAL: database "slave" does not exist
127.3.85.148:57777 - accepting connections
I20260619 18:43:08.113628 3414 mini_ranger.cc:161] Starting Ranger...
I20260619 18:43:08.141515 3414 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:43:08.728037 3414 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:43:08.728170 3414 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin
2026-06-19 18:43:09,002 [I] DB FLAVOR :POSTGRES
2026-06-19 18:43:09,002 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:43:09,003 [I] Checking connection..
2026-06-19 18:43:09,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:43:09,286 [I] Checking connection passed.
2026-06-19 18:43:09,287 [I] --------- Verifying version history table ---------
2026-06-19 18:43:09,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:09,550 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:43:09,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:09,812 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:43:09,812 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:43:09,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:43:10.090 UTC [15397] WARNING: there is no transaction in progress
2026-06-19 18:43:10,097 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:43:10,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:43:10,367 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:43:10,367 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:43:10,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:43:10,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:43:10,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-09fp.c.gcp-upstream.internal','N') ;"
2026-06-19 18:43:11,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:43:11,409 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:43:11,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:43:11,681 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:43:11,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:43:11,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:43:12,240 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:43:12,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:43:13.316 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.334 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.353 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.371 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.389 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.409 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.473 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.481 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.499 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.511 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.522 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.539 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.551 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.560 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.568 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:13.577 UTC [15505] WARNING: there is no transaction in progress
2026-06-19 18:43:14,406 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:43:14,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:43:14,663 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:43:14,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:43:14,934 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:43:14,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:43:15,182 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:43:15,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:43:15,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:43:15,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-09fp.c.gcp-upstream.internal';"
2026-06-19 18:43:15,966 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:43:15,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:43:16,218 [I] DB_PATCHES have already been applied
I20260619 18:43:16.224960 3414 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHiUM6J/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:43:16.225104 3414 mini_ranger.cc:193] Using host: 127.3.85.148
I20260619 18:43:16.241242 3414 mini_ranger.cc:227] Ranger admin URL: http://127.3.85.148:59883
Jun 19, 2026 6:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:43:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:43:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:43:17.400180 3414 mini_ranger.cc:160] Time spent starting Ranger: real 9.287s user 0.000s sys 0.044s
Jun 19, 2026 6:43:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:43:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:43:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:43:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:43:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:43:41.906780 3414 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.85.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.85.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangerkms_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.85.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/rangerkms_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.85.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.85.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.85.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/HTTP_127.3.85.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:43:42 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894622, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:43:42.085405 3414 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:43:42.110117 3414 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:43:43.113896 3414 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:43:43.114027 3414 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms
I20260619 18:43:43.114074 3414 mini_ranger_kms.cc:227] Using postgres at 127.3.85.148:57777
2026-06-19 18:43:43,480 [I] DB FLAVOR :POSTGRES
2026-06-19 18:43:43,481 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:43:43,481 [I] Checking connection
2026-06-19 18:43:43,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:43:43,739 [I] connection success
2026-06-19 18:43:43,739 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:43:43,739 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:43:43,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:43:43,987 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:43:43,987 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:43:43,987 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:43:43,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.85.148:57777/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:43:44,275 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:43:44.856225 3414 mini_ranger_kms.cc:310] Created kms service
I20260619 18:43:45.084414 3414 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:43:45.134341 3414 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:43:45 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894622, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:45.544793 3414 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:43:45.545130 3414 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHiUM6J/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHiUM6J/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:43:45.545212 3414 mini_ranger_kms.cc:240] Using host: 127.3.85.148
I20260619 18:43:45.564014 3414 mini_ranger_kms.cc:276] Ranger KMS PID: 15686
I20260619 18:43:45.564180 3414 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.85.148:36905
Jun 19, 2026 6:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:43:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:43:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:43:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:43:47.908870 3414 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.823s user 0.059s sys 0.327s
I20260619 18:43:47.909135 3414 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:43:47.909308 3414 mini_ranger_kms.cc:401] 127.3.85.148:36905/kms/v1/keys
Jun 19 18:43:48 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:43:48 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
Jun 19 18:43:48 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894628, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.85.148@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
Jun 19 18:43:51 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:43:51.376784 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38707
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:38707
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:51.779491 15736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:51.779899 15736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:51.780077 15736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:51.788125 15736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:51.788301 15736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:51.788414 15736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:51.788514 15736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:51.788605 15736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:51.799816 15736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.85.148:36905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:38707
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:51.801518 15736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:51.803277 15736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:51.818107 15741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:51.818106 15742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:51.818553 15736 server_base.cc:1034] running on GCE node
W20260619 18:43:51.818100 15744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:51.820030 15736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:51.822139 15736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:51.823415 15736 hybrid_clock.cc:648] HybridClock initialized: now 1781894631823317 us; error 91 us; skew 500 ppm
I20260619 18:43:51.823856 15736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:51 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:51.827939 15736 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:43:51.829941 15736 webserver.cc:460] Webserver started at http://127.3.85.190:40727/ using document root <none> and password file <none>
I20260619 18:43:51.830832 15736 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:51.830993 15736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:51.831343 15736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:51 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:51.898669 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "03bc8feb8d7642f4a0256efe611ccf9d"
format_stamp: "Formatted at 2026-06-19 18:43:51 on dist-test-slave-09fp"
server_key: "a29edba2fb125758d00028d9428ddaa0"
server_key_iv: "b47890822349842b4cd18cec849ebb19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:51.899694 15736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "03bc8feb8d7642f4a0256efe611ccf9d"
format_stamp: "Formatted at 2026-06-19 18:43:51 on dist-test-slave-09fp"
server_key: "a29edba2fb125758d00028d9428ddaa0"
server_key_iv: "b47890822349842b4cd18cec849ebb19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:51.905920 15736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:43:51.927657 15751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:51.929407 15736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:43:51.929652 15736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "03bc8feb8d7642f4a0256efe611ccf9d"
format_stamp: "Formatted at 2026-06-19 18:43:51 on dist-test-slave-09fp"
server_key: "a29edba2fb125758d00028d9428ddaa0"
server_key_iv: "b47890822349842b4cd18cec849ebb19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:51.929917 15736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:51.948108 15736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:51.952804 15736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:51.953183 15736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:51.974422 15736 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:38707
I20260619 18:43:51.974431 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:38707 every 8 connection(s)
I20260619 18:43:51.976009 15736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:51.980561 15803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:51.985844 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 15736
I20260619 18:43:51.986070 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:43:51.986915 3414 external_mini_cluster.cc:1428] Setting key 88b4f188d1387d72fa2a02f368a7f08a
I20260619 18:43:51.995919 15803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Bootstrap starting.
I20260619 18:43:52.001793 15803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:52.003742 15803 log.cc:824] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:52.007898 15803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: No bootstrap required, opened a new log
I20260619 18:43:52.018182 15803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:52.018807 15803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 18:43:52 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:52.018946 15803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bc8feb8d7642f4a0256efe611ccf9d, State: Initialized, Role: FOLLOWER
I20260619 18:43:52.019744 15803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:52.020047 15803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:52.020290 15803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:52.020530 15803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:52.023470 15803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:52.024096 15803 leader_election.cc:302] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bc8feb8d7642f4a0256efe611ccf9d; no voters:
I20260619 18:43:52.024845 15803 leader_election.cc:288] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:43:52.025911 15808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:52.027478 15803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:52.030073 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:51.989253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.3.85.190:38707)
0619 18:43:52.004209 (+ 14956us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:52.004228 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:52.004286 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:43:52.006798 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:52.006831 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:52.007367 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:52.007842 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:52.009582 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:52.011879 (+ 2297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:52.013827 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:52.015078 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:52.015298 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:52.020364 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:52.020379 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:52.020404 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:52.020478 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:52.024706 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:52.025744 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:52.025754 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:52.025764 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:52.025969 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:52.026471 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:52.026476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:52.026479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:52.027249 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:52.027554 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:52.028079 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 18:43:52.028573 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14680,"thread_start_us":14448,"threads_started":1}
I20260619 18:43:52.035128 15808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 LEADER]: Becoming Leader. State: Replica: 03bc8feb8d7642f4a0256efe611ccf9d, State: Running, Role: LEADER
I20260619 18:43:52.035907 15808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:52.044792 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } } }
I20260619 18:43:52.045408 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:52.044602 15810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bc8feb8d7642f4a0256efe611ccf9d. Latest consensus state: current_term: 1 leader_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } } }
I20260619 18:43:52.046831 15810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:52.052697 15817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:52.058890 15817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:52.059719 15803 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:43:52.093617 15817 catalog_manager.cc:1347] Generated new cluster ID: c88fc67287b7400db6ba9adc0a91e7c2
I20260619 18:43:52.093899 15817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:52.138474 15817 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:43:52.140539 15817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:52.152323 15817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Generated new TSK 0
I20260619 18:43:52.153555 15817 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:43:52.991426 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:43:53 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:43:53.483311 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:53.483857 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:53.484067 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:53.496747 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:53.497000 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:53.497234 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:43:53.509198 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:53.510689 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:53.512106 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:53.526826 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:53.526845 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:53.527596 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:53.528007 15848 server_base.cc:1034] running on GCE node
I20260619 18:43:53.528774 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:53.529942 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:53.531226 15848 hybrid_clock.cc:648] HybridClock initialized: now 1781894633531129 us; error 93 us; skew 500 ppm
I20260619 18:43:53.531668 15848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:53 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:53.536681 15848 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:43:53.551210 15848 webserver.cc:460] Webserver started at http://127.3.85.129:33183/ using document root <none> and password file <none>
I20260619 18:43:53.552265 15848 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:53.552474 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:53.552874 15848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:53 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:53.570032 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "ed73bf5e5c214e3998ac0dadfe48007c"
format_stamp: "Formatted at 2026-06-19 18:43:53 on dist-test-slave-09fp"
server_key: "624000370f52cf9a006d290db84b9d35"
server_key_iv: "0eff4372fa36922dce734fc006a80d19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:53.571390 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "ed73bf5e5c214e3998ac0dadfe48007c"
format_stamp: "Formatted at 2026-06-19 18:43:53 on dist-test-slave-09fp"
server_key: "624000370f52cf9a006d290db84b9d35"
server_key_iv: "0eff4372fa36922dce734fc006a80d19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:53.582598 15848 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.008s
I20260619 18:43:53.610317 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:53.612684 15848 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260619 18:43:53.612931 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "ed73bf5e5c214e3998ac0dadfe48007c"
format_stamp: "Formatted at 2026-06-19 18:43:53 on dist-test-slave-09fp"
server_key: "624000370f52cf9a006d290db84b9d35"
server_key_iv: "0eff4372fa36922dce734fc006a80d19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:53.613238 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:53.666317 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:53.672230 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:53.672667 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:53.673966 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:53.675940 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:53.676077 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 18:43:53.676221 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:53.676292 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:43:53 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:53.882694 15848 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:33301
I20260619 18:43:53.882706 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:33301 every 8 connection(s)
I20260619 18:43:53.884179 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:53.889161 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 15848
I20260619 18:43:53.889389 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:43:53.890321 3414 external_mini_cluster.cc:1428] Setting key 486a2a1d2578e5b02a4703279261b71f
Jun 19 18:43:53 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:53.926620 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:53.888459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.129:58079)
0619 18:43:53.899568 (+ 11109us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:53.899634 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:53.903323 (+ 3689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:53.904636 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:53.904653 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:53.905394 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:53.906414 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:53.906478 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:53.908283 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:53.908292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:53.910286 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:53.910331 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:53.911620 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:53.911626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:53.911938 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:53.912711 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:53.912736 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:53.915396 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:53.920232 (+ 4836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:53.920247 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:53.920271 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:53.921595 (+ 1324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:53.922203 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:53.922211 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:53.922217 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:53.922593 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:53.923372 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:53.923384 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:53.923926 (+ 542us) client_negotiation.cc:769] Sending connection context
0619 18:43:53.924253 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 18:43:53.924927 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10741,"thread_start_us":10480,"threads_started":1}
I20260619 18:43:53.928241 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:53.888138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:58079 (local address 127.3.85.190:38707)
0619 18:43:53.898924 (+ 10786us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:53.898931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:53.899762 (+ 831us) server_negotiation.cc:407] Connection header received
0619 18:43:53.903743 (+ 3981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:53.903752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:53.903869 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:53.903961 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:53.906772 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:53.908019 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:53.910562 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:53.911351 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:53.911507 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:53.915753 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:53.915762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:53.915770 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:53.915823 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:53.919968 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:53.921852 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:53.921861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:53.921867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:53.922056 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:53.922791 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:53.922799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:53.922806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:53.923216 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:53.927183 (+ 3967us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:53.927407 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:43:53.927726 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":10778,"threads_started":1}
I20260619 18:43:53.930188 15981 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
I20260619 18:43:53.930614 15981 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:53.931679 15981 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:43:53.934710 15768 ts_manager.cc:194] Registered new tserver with Master: ed73bf5e5c214e3998ac0dadfe48007c (127.3.85.129:33301)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:43:53.937109 15768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:58079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:43:53.967136 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:43:54 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894633, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
W20260619 18:43:54.452968 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:54.453347 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:54.453517 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:54.461222 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:54.461419 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:54.461611 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:43:54.472222 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:54.473698 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:54.475346 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:54.496915 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:54.497798 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:54.498253 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:54.499596 15987 server_base.cc:1034] running on GCE node
I20260619 18:43:54.500372 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:54.501521 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:54.502974 15987 hybrid_clock.cc:648] HybridClock initialized: now 1781894634502749 us; error 62 us; skew 500 ppm
I20260619 18:43:54.503424 15987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:54 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:54.507063 15987 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:43:54.509055 15987 webserver.cc:460] Webserver started at http://127.3.85.130:38571/ using document root <none> and password file <none>
I20260619 18:43:54.509892 15987 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:54.510007 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:54.510407 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:54 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:54.529052 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "a07fb89f73a54f18b01a637e58df4e17"
format_stamp: "Formatted at 2026-06-19 18:43:54 on dist-test-slave-09fp"
server_key: "5505b5b2d14fef869022f6621a6b436f"
server_key_iv: "e319f7aa4c5e337c1feb49cb45f61eb6"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:54.530489 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "a07fb89f73a54f18b01a637e58df4e17"
format_stamp: "Formatted at 2026-06-19 18:43:54 on dist-test-slave-09fp"
server_key: "5505b5b2d14fef869022f6621a6b436f"
server_key_iv: "e319f7aa4c5e337c1feb49cb45f61eb6"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:54.538703 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260619 18:43:54.561273 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:54.563486 15987 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 18:43:54.563809 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "a07fb89f73a54f18b01a637e58df4e17"
format_stamp: "Formatted at 2026-06-19 18:43:54 on dist-test-slave-09fp"
server_key: "5505b5b2d14fef869022f6621a6b436f"
server_key_iv: "e319f7aa4c5e337c1feb49cb45f61eb6"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:54.564188 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:54.580545 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:54.585137 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:54.585484 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:54.586634 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:54.588249 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:54.588367 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:54.588531 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:54.588609 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:54.636844 15987 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:41895
I20260619 18:43:54.636866 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:41895 every 8 connection(s)
I20260619 18:43:54.638523 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:54.641570 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 15987
I20260619 18:43:54.641798 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:43:54.642725 3414 external_mini_cluster.cc:1428] Setting key 7f2f9f98fb65c5acba08dc4830416945
Jun 19 18:43:54 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894634, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:54.676741 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:54.646297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51311 (local address 127.3.85.190:38707)
0619 18:43:54.651234 (+ 4937us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:54.651239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:54.651255 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:43:54.651307 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:54.651312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:54.652139 (+ 827us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:54.652255 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:54.657003 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:54.658211 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:54.661194 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:54.661935 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:54.662129 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:54.667199 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:54.667209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:54.667217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:54.667302 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:54.671835 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:54.673136 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:54.673146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:54.673153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:54.673352 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:54.674029 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:54.674035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:54.674038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:54.674545 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:54.674717 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:54.675984 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0619 18:43:54.676298 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4804,"thread_start_us":4688,"threads_started":1}
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:43:54.679608 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:54.645669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.130:51311)
0619 18:43:54.646841 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:54.646905 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:54.650256 (+ 3351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:54.654880 (+ 4624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:54.654898 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:54.655646 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:54.656674 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:54.656711 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:54.658497 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:54.658506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:54.660898 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:54.660935 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:54.662243 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:54.662251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:54.662672 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:54.663568 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:54.663606 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:54.666842 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:54.672125 (+ 5283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:54.672138 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:54.672163 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:54.672936 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:54.673529 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:54.673536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:54.673540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:54.673865 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:54.674836 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:54.674846 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:54.675444 (+ 598us) client_negotiation.cc:769] Sending connection context
0619 18:43:54.677413 (+ 1969us) client_negotiation.cc:240] Negotiation successful
0619 18:43:54.678103 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":614,"threads_started":1}
I20260619 18:43:54.682216 16124 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
I20260619 18:43:54.682659 16124 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:54.683722 16124 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:54.686625 15768 ts_manager.cc:194] Registered new tserver with Master: a07fb89f73a54f18b01a637e58df4e17 (127.3.85.130:41895)
I20260619 18:43:54.687989 15768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:51311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:43:54.715420 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:43:54.941855 15981 heartbeater.cc:503] Master 127.3.85.190:38707 was elected leader, sending a full tablet report...
W20260619 18:43:55.083168 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:55.083479 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:55.083575 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:55.091219 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:55.091346 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:55.091460 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:43:55.101783 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:55.103129 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:55.104542 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:55.118824 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:55.118800 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:55.118847 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:55.119869 16130 server_base.cc:1034] running on GCE node
I20260619 18:43:55.120589 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:55.121646 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:55.122898 16130 hybrid_clock.cc:648] HybridClock initialized: now 1781894635122847 us; error 49 us; skew 500 ppm
I20260619 18:43:55.123245 16130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:55.126744 16130 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:43:55.128413 16130 webserver.cc:460] Webserver started at http://127.3.85.131:40599/ using document root <none> and password file <none>
I20260619 18:43:55.129215 16130 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:55.129328 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:43:55.129592 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:55.145529 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "131ccbc73d5343c1a19faa80f6f8e72f"
format_stamp: "Formatted at 2026-06-19 18:43:55 on dist-test-slave-09fp"
server_key: "99bc8ac5f82b30a94de0304b5a923984"
server_key_iv: "2f060f2161c7e2abfc9b787e166f8783"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:55.146709 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "131ccbc73d5343c1a19faa80f6f8e72f"
format_stamp: "Formatted at 2026-06-19 18:43:55 on dist-test-slave-09fp"
server_key: "99bc8ac5f82b30a94de0304b5a923984"
server_key_iv: "2f060f2161c7e2abfc9b787e166f8783"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:55.154844 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 18:43:55.174175 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:55.176395 16130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:43:55.176739 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "131ccbc73d5343c1a19faa80f6f8e72f"
format_stamp: "Formatted at 2026-06-19 18:43:55 on dist-test-slave-09fp"
server_key: "99bc8ac5f82b30a94de0304b5a923984"
server_key_iv: "2f060f2161c7e2abfc9b787e166f8783"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:55.177019 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:55.199473 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:55.204394 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:55.204771 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:55.205839 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:55.207505 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:55.207639 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:55.207932 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:55.208041 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:55.245702 16130 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:43531
I20260619 18:43:55.245751 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:43531 every 8 connection(s)
I20260619 18:43:55.247092 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:55.253340 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16130
I20260619 18:43:55.253520 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:43:55.254215 3414 external_mini_cluster.cc:1428] Setting key b396a0efd2011a8367ca1a6170b813ae
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:55.268946 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:55.249879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:49355 (local address 127.3.85.190:38707)
0619 18:43:55.250575 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:55.250582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:55.251577 (+ 995us) server_negotiation.cc:407] Connection header received
0619 18:43:55.253402 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:55.253409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:55.253507 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:55.253645 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:55.255909 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.256749 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.258591 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.259042 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.259156 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:55.262658 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:55.262664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:55.262668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:55.262705 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:55.265558 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.266381 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.266385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.266387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.266593 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.266993 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.266996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.266999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.267286 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:55.267397 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:55.268240 (+ 843us) server_negotiation.cc:299] Negotiation successful
0619 18:43:55.268520 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":285,"threads_started":1}
I20260619 18:43:55.270042 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:55.250563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.131:49355)
0619 18:43:55.251376 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:55.251424 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:55.253079 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:55.254161 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:55.254174 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:55.254841 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:55.255699 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:55.255732 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.256919 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.256923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:55.258358 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:55.258374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.259282 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.259287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:55.259516 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:55.260228 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:55.260252 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:55.262336 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:55.265752 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:55.265760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:55.265775 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:55.266240 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:55.266699 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:55.266702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:55.266704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:55.266886 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:55.267396 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:55.267401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:55.267830 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 18:43:55.268060 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 18:43:55.268587 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":326,"threads_started":1}
I20260619 18:43:55.271620 16258 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
I20260619 18:43:55.271974 16258 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:55.272744 16258 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:55.274317 15768 ts_manager.cc:194] Registered new tserver with Master: 131ccbc73d5343c1a19faa80f6f8e72f (127.3.85.131:43531)
I20260619 18:43:55.275238 15768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:49355
I20260619 18:43:55.283506 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:43:55.285315 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 15848
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:43:55.304181 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:55.286427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.3.85.129:33301)
0619 18:43:55.287160 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:55.287168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:55.287199 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:43:55.287270 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:55.287276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:55.287502 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:55.287633 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:55.289173 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.290244 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.291808 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.292402 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.292510 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:55.297328 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:55.297345 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:55.297362 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:55.297408 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:55.300758 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.301727 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.301736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.301739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.301915 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.302375 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.302379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.302382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.302893 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:55.303167 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:55.303501 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 18:43:55.303736 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":392,"threads_started":1}
I20260619 18:43:55.305585 15976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:55.393034 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 15848
I20260619 18:43:55.409694 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 15987
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:43:55.427667 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:55.410606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.3.85.130:41895)
0619 18:43:55.411161 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:55.411168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:55.411193 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:43:55.411256 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:55.411262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:55.411479 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:55.411674 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:55.412955 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.413975 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.415526 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.416119 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.416285 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:55.421175 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:55.421191 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:55.421208 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:55.421248 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:55.424338 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.425422 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.425430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.425432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.425587 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.426011 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.426015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.426017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.426392 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:55.426730 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:55.427014 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 18:43:55.427230 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
I20260619 18:43:55.429108 16119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:55.513275 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 15987
I20260619 18:43:55.530040 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16130
Jun 19 18:43:55 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:43:55.548763 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:55.531124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.3.85.131:43531)
0619 18:43:55.531769 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:55.531776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:55.531806 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:43:55.531868 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:55.531874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:55.532098 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:55.532256 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:55.533752 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.535014 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.536472 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:55.537077 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:55.537260 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:55.542276 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:55.542295 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:55.542312 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:55.542356 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:55.545525 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.546414 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.546424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.546428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.546623 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:55.547018 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:55.547025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:55.547028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:55.547396 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:55.547728 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:55.548091 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:43:55.548378 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":320,"threads_started":1}
I20260619 18:43:55.550180 16253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:55.635315 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16130
I20260619 18:43:55.651502 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 15736
I20260619 18:43:55.652887 15798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:55.762840 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 15736
I20260619 18:43:55.784873 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38707
--webserver_interface=127.3.85.190
--webserver_port=40727
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:38707
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:43:56.129930 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:56.130249 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:56.130340 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:56.137943 16270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:43:56.138067 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:56.138115 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:56.138164 16270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:43:56.138212 16270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:43:56.148839 16270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:38707
--ranger_config_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:38707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=40727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:56.150128 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:56.151530 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:56.164317 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:56.164284 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:56.164368 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:56.165719 16270 server_base.cc:1034] running on GCE node
I20260619 18:43:56.166685 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:56.168308 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:56.169540 16270 hybrid_clock.cc:648] HybridClock initialized: now 1781894636169472 us; error 67 us; skew 500 ppm
I20260619 18:43:56.170061 16270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:56 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:56.173810 16270 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:43:56.175695 16270 webserver.cc:460] Webserver started at http://127.3.85.190:40727/ using document root <none> and password file <none>
I20260619 18:43:56.176414 16270 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:56.176538 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:43:56 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:56.233990 16270 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 18:43:56.237551 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:56.239295 16270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:43:56.239537 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "03bc8feb8d7642f4a0256efe611ccf9d"
format_stamp: "Formatted at 2026-06-19 18:43:51 on dist-test-slave-09fp"
server_key: "a29edba2fb125758d00028d9428ddaa0"
server_key_iv: "b47890822349842b4cd18cec849ebb19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:56.240605 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:56.273051 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:56.277559 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:56.277909 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:56.298352 16270 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:38707
I20260619 18:43:56.298408 16337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:38707 every 8 connection(s)
I20260619 18:43:56.299968 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:43:56.301182 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16270
I20260619 18:43:56.302171 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:33301
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:38707
--webserver_port=33183
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:43:56.307725 16338 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:43:56.311436 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Bootstrap starting.
I20260619 18:43:56.331050 16338 log.cc:824] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:56.340551 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:43:56.341149 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Bootstrap complete.
I20260619 18:43:56.351042 16338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:56.351997 16338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bc8feb8d7642f4a0256efe611ccf9d, State: Initialized, Role: FOLLOWER
I20260619 18:43:56.352799 16338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:56.353070 16338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:43:56.353189 16338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:43:56.353415 16338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:43:56.357626 16338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:56.358273 16338 leader_election.cc:302] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03bc8feb8d7642f4a0256efe611ccf9d; no voters:
I20260619 18:43:56.359056 16338 leader_election.cc:288] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:43:56.359200 16342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:43:56.360423 16342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [term 2 LEADER]: Becoming Leader. State: Replica: 03bc8feb8d7642f4a0256efe611ccf9d, State: Running, Role: LEADER
I20260619 18:43:56.361096 16342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } }
I20260619 18:43:56.361457 16338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:43:56.363858 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03bc8feb8d7642f4a0256efe611ccf9d. Latest consensus state: current_term: 2 leader_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } } }
I20260619 18:43:56.364385 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:56.365037 16343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03bc8feb8d7642f4a0256efe611ccf9d" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 38707 } } }
I20260619 18:43:56.365356 16343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d [sys.catalog]: This master's current role is: LEADER
I20260619 18:43:56.368007 16348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:43:56.374681 16348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:43:56.377377 16348 catalog_manager.cc:1259] Loaded cluster ID: c88fc67287b7400db6ba9adc0a91e7c2
I20260619 18:43:56.377851 16348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:43:56.382898 16348 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:43:56.384822 16348 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Loaded TSK: 0
I20260619 18:43:56.387032 16348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:43:56.391633 16338 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 18:43:56.773800 16340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:56.774226 16340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:56.774407 16340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:56.785930 16340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:56.786103 16340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:56.786316 16340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:43:56.802884 16340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:33301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=33183
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:56.804718 16340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:56.806181 16340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:56.825888 16377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:56.826226 16376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:56.831176 16379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:56.832015 16340 server_base.cc:1034] running on GCE node
I20260619 18:43:56.832940 16340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:56.841605 16340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:56.842933 16340 hybrid_clock.cc:648] HybridClock initialized: now 1781894636842860 us; error 86 us; skew 500 ppm
I20260619 18:43:56.843389 16340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:56 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:56.852243 16340 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:43:56.854609 16340 webserver.cc:460] Webserver started at http://127.3.85.129:33183/ using document root <none> and password file <none>
I20260619 18:43:56.855722 16340 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:56.855904 16340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:43:56 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:56.901906 16340 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.006s sys 0.000s
I20260619 18:43:56.906591 16387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:56.908720 16340 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:43:56.908943 16340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "ed73bf5e5c214e3998ac0dadfe48007c"
format_stamp: "Formatted at 2026-06-19 18:43:53 on dist-test-slave-09fp"
server_key: "624000370f52cf9a006d290db84b9d35"
server_key_iv: "0eff4372fa36922dce734fc006a80d19"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:56.910545 16340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:56.946440 16340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:56.952795 16340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:56.953164 16340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:56.964632 16340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:56.966642 16340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:56.966779 16340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:56.966897 16340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:56.966972 16340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:57.020892 16340 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:33301
I20260619 18:43:57.020921 16499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:33301 every 8 connection(s)
I20260619 18:43:57.022996 16340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:43:57.024989 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16340
I20260619 18:43:57.025846 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:41895
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:38707
--webserver_port=38571
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:43:57 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:57.068831 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:57.027022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.129:44549)
0619 18:43:57.038630 (+ 11608us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:57.038694 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:57.041455 (+ 2761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:57.044345 (+ 2890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:57.044358 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:57.045054 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:57.045979 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:57.046017 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.048760 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.048769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:57.051000 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:57.051024 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.053140 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.053149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:57.053438 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:57.054376 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:57.054410 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:57.057403 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:57.062839 (+ 5436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:57.062849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:57.062866 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:57.063521 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:57.064491 (+ 970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:57.064496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:57.064500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:57.064776 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:57.065966 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:57.065977 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:57.066409 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 18:43:57.066826 (+ 417us) client_negotiation.cc:240] Negotiation successful
0619 18:43:57.067412 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11253,"thread_start_us":11070,"threads_started":1}
I20260619 18:43:57.070354 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:57.027110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:44549 (local address 127.3.85.190:38707)
0619 18:43:57.038151 (+ 11041us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:57.038178 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:57.039265 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 18:43:57.042326 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:57.042353 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:57.043004 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:57.043474 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:57.046311 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.048475 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.051261 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.052437 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.052697 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:57.057859 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:57.057874 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:57.057896 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:57.057950 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:57.062590 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:57.064075 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:57.064086 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:57.064094 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:57.064324 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:57.064929 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:57.064935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:57.064938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:57.065701 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:57.065913 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:57.067143 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0619 18:43:57.067569 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10717,"thread_start_us":10429,"threads_started":1}
I20260619 18:43:57.074067 16500 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
I20260619 18:43:57.074434 16500 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:57.075361 16500 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
W20260619 18:43:57.078853 16500 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:38707 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:43:57.079144 16500 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:57.079499 16500 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:57.081039 16500 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:57.081367 16500 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
W20260619 18:43:57.082736 16500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:43:57.463738 16504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:57.464120 16504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:57.464299 16504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:57.475173 16504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:57.475411 16504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:57.475633 16504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:43:57.492444 16504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:41895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=38571
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:57.494635 16504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:57.496454 16504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:57.514035 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:57.514056 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:57.514544 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:57.516158 16504 server_base.cc:1034] running on GCE node
I20260619 18:43:57.517323 16504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:43:57.518839 16504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:57.520124 16504 hybrid_clock.cc:648] HybridClock initialized: now 1781894637520045 us; error 77 us; skew 500 ppm
I20260619 18:43:57.520586 16504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:57 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:57.525467 16504 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:43:57.528076 16504 webserver.cc:460] Webserver started at http://127.3.85.130:38571/ using document root <none> and password file <none>
I20260619 18:43:57.529084 16504 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:57.529259 16504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:43:57 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:57.554590 16504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:43:57.557631 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:57.559327 16504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:43:57.559540 16504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "a07fb89f73a54f18b01a637e58df4e17"
format_stamp: "Formatted at 2026-06-19 18:43:54 on dist-test-slave-09fp"
server_key: "5505b5b2d14fef869022f6621a6b436f"
server_key_iv: "e319f7aa4c5e337c1feb49cb45f61eb6"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:57.560588 16504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:57.586143 16504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:57.591110 16504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:57.591441 16504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:57.592563 16504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:57.594167 16504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:57.594297 16504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:57.594412 16504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:57.594511 16504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 18:43:57.701705 16504 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:41895
I20260619 18:43:57.701745 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:41895 every 8 connection(s)
I20260619 18:43:57.703799 16504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:43:57.713838 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16504
I20260619 18:43:57.714618 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.85.148:36905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:43531
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:38707
--webserver_port=40599
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:41477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:43:57 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
Jun 19 18:43:57 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:57.742611 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:57.710829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:36001 (local address 127.3.85.190:38707)
0619 18:43:57.721333 (+ 10504us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:57.721340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:57.721368 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:43:57.721444 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:57.721450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:57.721546 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:57.721695 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:57.724335 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.725539 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.728194 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.728899 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.729052 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:57.733978 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:57.733989 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:57.733994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:57.734042 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:57.738036 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:57.739364 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:57.739372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:57.739375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:57.739552 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:57.740209 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:57.740215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:57.740219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:57.740649 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:57.740811 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:57.741936 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0619 18:43:57.742207 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10360,"thread_start_us":10224,"threads_started":1}
I20260619 18:43:57.745410 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:57.710764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.130:36001)
0619 18:43:57.713439 (+ 2675us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:57.713503 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:57.719233 (+ 5730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:57.722294 (+ 3061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:57.722310 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:57.723066 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:57.724031 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:57.724065 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.725843 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.725849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:57.727955 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:57.727978 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:57.729435 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:57.729445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:57.729796 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:57.730788 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:57.730845 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:57.733593 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:57.738334 (+ 4741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:57.738345 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:57.738368 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:57.739115 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:57.739743 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:57.739748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:57.739752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:57.740057 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:57.740927 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:57.740937 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:57.741451 (+ 514us) client_negotiation.cc:769] Sending connection context
0619 18:43:57.742976 (+ 1525us) client_negotiation.cc:240] Negotiation successful
0619 18:43:57.743621 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2307,"thread_start_us":2032,"threads_started":1}
I20260619 18:43:57.747993 16645 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
I20260619 18:43:57.748394 16645 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:57.749344 16645 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:57.751839 16303 ts_manager.cc:194] Registered new tserver with Master: a07fb89f73a54f18b01a637e58df4e17 (127.3.85.130:41895)
I20260619 18:43:57.754283 16303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:36001
I20260619 18:43:58.082919 16500 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:58.083411 16500 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:58.085175 16303 ts_manager.cc:194] Registered new tserver with Master: ed73bf5e5c214e3998ac0dadfe48007c (127.3.85.129:33301)
I20260619 18:43:58.086402 16303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:44549
W20260619 18:43:58.174079 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:43:58.174650 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:43:58.174831 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:43:58.187104 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:43:58.187280 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:43:58.187558 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:43:58.204238 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:41477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.85.148:36905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:43531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=40599
--tserver_master_addrs=127.3.85.190:38707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:43:58.206343 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:43:58.208285 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:43:58.237362 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:43:58.237612 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:58.247541 16649 server_base.cc:1034] running on GCE node
W20260619 18:43:58.248001 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:43:58.248895 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Jun 19 18:43:58 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:58.250471 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:43:58.251766 16649 hybrid_clock.cc:648] HybridClock initialized: now 1781894638251683 us; error 57 us; skew 500 ppm
I20260619 18:43:58.252177 16649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:43:58 dist-test-slave-09fp krb5kdc[15292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:43:58.257783 16649 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:43:58.283408 16649 webserver.cc:460] Webserver started at http://127.3.85.131:40599/ using document root <none> and password file <none>
I20260619 18:43:58.284746 16649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:43:58.284976 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:43:58 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:58.329402 16649 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:43:58.345081 16666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:58.347427 16649 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.012s
I20260619 18:43:58.347736 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "131ccbc73d5343c1a19faa80f6f8e72f"
format_stamp: "Formatted at 2026-06-19 18:43:55 on dist-test-slave-09fp"
server_key: "99bc8ac5f82b30a94de0304b5a923984"
server_key_iv: "2f060f2161c7e2abfc9b787e166f8783"
server_key_version: "kuduclusterkey@0"
I20260619 18:43:58.349242 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:43:58.366925 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:43:58.372447 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:43:58.372771 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:43:58.374038 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:43:58.375922 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:43:58.376031 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:58.376129 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:43:58.376197 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:43:58.424223 16649 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:43531
I20260619 18:43:58.424415 16778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:43531 every 8 connection(s)
I20260619 18:43:58.426419 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:43:58.426723 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16649
Jun 19 18:43:58 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894638, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:43:58.466341 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.427908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.3.85.190:38707)
0619 18:43:58.443438 (+ 15530us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.443448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.446698 (+ 3250us) server_negotiation.cc:407] Connection header received
0619 18:43:58.446804 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.446813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.446964 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.447100 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:58.452951 (+ 5851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.453703 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.455706 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.456246 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.456400 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.459726 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:58.459732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:58.459736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:58.459783 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:58.463159 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.464108 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.464114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.464116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.464256 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.464700 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.464711 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.464713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.465013 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:58.465308 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.465681 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.465943 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15340,"thread_start_us":15536,"threads_started":1}
I20260619 18:43:58.476944 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.437828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:42479 (local address 127.3.85.190:38707)
0619 18:43:58.446925 (+ 9097us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.446932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.450511 (+ 3579us) server_negotiation.cc:407] Connection header received
0619 18:43:58.451350 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.451357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.451478 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.451578 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:58.454339 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.455864 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.458764 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.459495 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.459671 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.470548 (+ 10877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:58.470554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:58.470558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:58.470594 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:58.473550 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.474399 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.474405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.474414 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.474601 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.475090 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.475096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.475099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.475390 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:58.475556 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.476318 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.476547 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8927,"thread_start_us":8771,"threads_started":1}
I20260619 18:43:58.478111 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.437212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:38707 (local address 127.3.85.131:42479)
0619 18:43:58.447924 (+ 10712us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.447987 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.450934 (+ 2947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.452303 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.452317 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.453027 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:43:58.453992 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.454026 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.456169 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.456175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.458533 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.458565 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.460118 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.460127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.460451 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.461186 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:43:58.461225 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:43:58.470104 (+ 8879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:43:58.473733 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:58.473744 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:58.473775 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:58.474249 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:58.474726 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:43:58.474732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:43:58.474735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:43:58.474956 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:43:58.475540 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:43:58.475549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:43:58.475938 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.476220 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.476802 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10364,"thread_start_us":10121,"threads_started":1}
I20260619 18:43:58.479755 16779 heartbeater.cc:347] Connected to a master server at 127.3.85.190:38707
I20260619 18:43:58.480078 16779 heartbeater.cc:464] Registering TS with master...
I20260619 18:43:58.480751 16779 heartbeater.cc:511] Master 127.3.85.190:38707 requested a full tablet report, sending...
I20260619 18:43:58.482275 16302 ts_manager.cc:194] Registered new tserver with Master: 131ccbc73d5343c1a19faa80f6f8e72f (127.3.85.131:43531)
I20260619 18:43:58.483343 16302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:42479
I20260619 18:43:58.486405 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:43:58.505777 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.490205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.3.85.190:38707)
0619 18:43:58.490400 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.490407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.490545 (+ 138us) server_negotiation.cc:407] Connection header received
0619 18:43:58.490906 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.490913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.491001 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.491060 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:58.492745 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.493476 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.495006 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.495445 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.495621 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.498390 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:58.498396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:58.498399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:58.498437 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:58.502034 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.503120 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.503127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.503131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.503304 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:58.503815 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:58.503821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:58.503824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:58.504232 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:58.504485 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.505138 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.505422 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:43:58.512671 16302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:43:58.520915 16302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:43:58.571461 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.538399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:41895 (local address 127.0.0.1:57616)
0619 18:43:58.554058 (+ 15659us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.554111 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.554336 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.557067 (+ 2731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.557079 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.557125 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:58.557593 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.557610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.562072 (+ 4462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.562082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.565034 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.565064 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.570064 (+ 5000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.570075 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.570385 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.570490 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.570668 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.570973 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15380,"thread_start_us":15210,"threads_started":1}
I20260619 18:43:58.572602 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.539023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.3.85.130:41895)
0619 18:43:58.552841 (+ 13818us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.552855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.554537 (+ 1682us) server_negotiation.cc:407] Connection header received
0619 18:43:58.555906 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.555919 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.556276 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.556564 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:58.557825 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.561808 (+ 3983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.565573 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.567495 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.567677 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.567958 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.571935 (+ 3977us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.572193 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13628,"thread_start_us":13481,"threads_started":1}
I20260619 18:43:58.579634 16580 tablet_service.cc:1467] Processing CreateTablet for tablet 75106f93aeac484697a83eec69d563fd (DEFAULT_TABLE table=test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:58.582345 16580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75106f93aeac484697a83eec69d563fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:58.601658 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.560828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33301 (local address 127.0.0.1:39912)
0619 18:43:58.574107 (+ 13279us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.574126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.574234 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.587439 (+ 13205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.587447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.587493 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:58.587810 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.587821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.593464 (+ 5643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.593473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.595617 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.595642 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.600741 (+ 5099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.600750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.600952 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.600992 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.601086 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.601250 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13156,"thread_start_us":13681,"threads_started":1}
I20260619 18:43:58.608276 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.544434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:43531 (local address 127.0.0.1:35490)
0619 18:43:58.573366 (+ 28932us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.573391 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.573544 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.595904 (+ 22360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.595910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.595954 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:58.596156 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.596170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.602063 (+ 5893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.602069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.604019 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.604042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.607308 (+ 3266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.607315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.607695 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.607748 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.607827 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.607988 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28762,"thread_start_us":29371,"threads_started":1}
I20260619 18:43:58.609313 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.555133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.3.85.129:33301)
0619 18:43:58.575237 (+ 20104us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.575247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.575277 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:43:58.575377 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.575386 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.575658 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.591288 (+ 15630us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:58.591623 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.593229 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.598582 (+ 5353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.600136 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.600299 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.600566 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.606682 (+ 6116us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.606934 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19926,"thread_start_us":20744,"threads_started":1}
I20260619 18:43:58.610564 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.545310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.3.85.131:43531)
0619 18:43:58.590619 (+ 45309us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.590628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.590668 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:43:58.590782 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.590790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.591082 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.596411 (+ 5329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:58.596753 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.601880 (+ 5127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.604324 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.605906 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.608761 (+ 2855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.609705 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.609802 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.610164 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45048,"thread_start_us":41914,"threads_started":1}
Jun 19 18:43:58 dist-test-slave-09fp krb5kdc[15292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894637, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for HTTP/127.3.85.148@KRBTEST.COM
I20260619 18:43:58.613997 16435 tablet_service.cc:1467] Processing CreateTablet for tablet 75106f93aeac484697a83eec69d563fd (DEFAULT_TABLE table=test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:58.615782 16435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75106f93aeac484697a83eec69d563fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:58.615388 16714 tablet_service.cc:1467] Processing CreateTablet for tablet 75106f93aeac484697a83eec69d563fd (DEFAULT_TABLE table=test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:43:58.617025 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75106f93aeac484697a83eec69d563fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:43:58.622747 16801 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Bootstrap starting.
I20260619 18:43:58.627218 16801 tablet_bootstrap.cc:654] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:58.628934 16801 log.cc:824] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:58.677093 16803 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Bootstrap starting.
I20260619 18:43:58.678622 16801 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: No bootstrap required, opened a new log
I20260619 18:43:58.679028 16801 ts_tablet_manager.cc:1397] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.044s
I20260619 18:43:58.683459 16803 tablet_bootstrap.cc:654] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:58.690150 16801 raft_consensus.cc:348] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.690917 16801 raft_consensus.cc:374] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:58.691087 16801 raft_consensus.cc:729] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07fb89f73a54f18b01a637e58df4e17, State: Initialized, Role: FOLLOWER
I20260619 18:43:58.691957 16801 consensus_queue.cc:260] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.694815 16804 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Bootstrap starting.
I20260619 18:43:58.694823 16803 log.cc:824] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:58.699314 16645 heartbeater.cc:503] Master 127.3.85.190:38707 was elected leader, sending a full tablet report...
I20260619 18:43:58.700162 16801 ts_tablet_manager.cc:1428] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Time spent starting tablet: real 0.021s user 0.013s sys 0.003s
I20260619 18:43:58.700665 16804 tablet_bootstrap.cc:654] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Neither blocks nor log segments found. Creating new log.
I20260619 18:43:58.702085 16804 log.cc:824] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:43:58.706017 16804 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: No bootstrap required, opened a new log
I20260619 18:43:58.718685 16803 tablet_bootstrap.cc:492] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: No bootstrap required, opened a new log
I20260619 18:43:58.719182 16803 ts_tablet_manager.cc:1397] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.000s
I20260619 18:43:58.720093 16804 ts_tablet_manager.cc:1397] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
W20260619 18:43:58.723227 16646 tablet.cc:2369] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:58.742574 16803 raft_consensus.cc:348] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.743251 16803 raft_consensus.cc:374] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:58.743378 16803 raft_consensus.cc:729] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed73bf5e5c214e3998ac0dadfe48007c, State: Initialized, Role: FOLLOWER
I20260619 18:43:58.744323 16803 consensus_queue.cc:260] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.746771 16804 raft_consensus.cc:348] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.747373 16804 raft_consensus.cc:374] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:43:58.747521 16804 raft_consensus.cc:729] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131ccbc73d5343c1a19faa80f6f8e72f, State: Initialized, Role: FOLLOWER
I20260619 18:43:58.748387 16804 consensus_queue.cc:260] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.766541 16500 heartbeater.cc:503] Master 127.3.85.190:38707 was elected leader, sending a full tablet report...
I20260619 18:43:58.771548 16803 ts_tablet_manager.cc:1428] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Time spent starting tablet: real 0.052s user 0.016s sys 0.006s
I20260619 18:43:58.776736 16779 heartbeater.cc:503] Master 127.3.85.190:38707 was elected leader, sending a full tablet report...
I20260619 18:43:58.777521 16804 ts_tablet_manager.cc:1428] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Time spent starting tablet: real 0.057s user 0.027s sys 0.010s
W20260619 18:43:58.778056 16501 tablet.cc:2369] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:58.880223 16809 raft_consensus.cc:482] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:43:58.880815 16809 raft_consensus.cc:504] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.883286 16809 leader_election.cc:288] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07fb89f73a54f18b01a637e58df4e17 (127.3.85.130:41895), ed73bf5e5c214e3998ac0dadfe48007c (127.3.85.129:33301)
I20260619 18:43:58.896339 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.886884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:41895 (local address 127.3.85.131:44527)
0619 18:43:58.887112 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.887134 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.887283 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.888958 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.888966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.889003 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:58.889298 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.889311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.890899 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.890905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.893180 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.893198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.895533 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.895539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.895757 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.895793 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.895891 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.896044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":19}
I20260619 18:43:58.897117 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.888232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:44527 (local address 127.3.85.130:41895)
0619 18:43:58.888414 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.888420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.888439 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:43:58.888526 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.888535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.888644 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.888751 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:58.889500 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.890714 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.893436 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.895192 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.895342 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.895428 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.896714 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.896871 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:43:58.898600 16600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75106f93aeac484697a83eec69d563fd" candidate_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07fb89f73a54f18b01a637e58df4e17" is_pre_election: true
I20260619 18:43:58.898940 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.887757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:39107 (local address 127.3.85.129:33301)
0619 18:43:58.887959 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.887964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.887987 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:43:58.888266 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.888272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.888353 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.888415 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:43:58.889271 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.890407 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.893191 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.894478 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.894640 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.894732 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.898534 (+ 3802us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.898697 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:43:58.899255 16600 raft_consensus.cc:2454] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131ccbc73d5343c1a19faa80f6f8e72f in term 0.
I20260619 18:43:58.900228 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.887290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33301 (local address 127.3.85.131:39107)
0619 18:43:58.887899 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 18:43:58.887918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:43:58.888089 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:43:58.888644 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:43:58.888648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:43:58.888675 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:43:58.889058 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.889075 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.890650 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.890655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.892965 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:43:58.892984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.895533 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.895538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:43:58.898253 (+ 2715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.898298 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:43:58.899347 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0619 18:43:58.899588 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":5,"thread_start_us":1003,"threads_started":1}
I20260619 18:43:58.900152 16669 leader_election.cc:302] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131ccbc73d5343c1a19faa80f6f8e72f, a07fb89f73a54f18b01a637e58df4e17; no voters:
I20260619 18:43:58.900887 16809 raft_consensus.cc:2781] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:43:58.901057 16809 raft_consensus.cc:482] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:43:58.901160 16809 raft_consensus.cc:3037] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:58.901814 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75106f93aeac484697a83eec69d563fd" candidate_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" is_pre_election: true
I20260619 18:43:58.902369 16455 raft_consensus.cc:2454] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131ccbc73d5343c1a19faa80f6f8e72f in term 0.
I20260619 18:43:58.904470 16809 raft_consensus.cc:504] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.905222 16809 leader_election.cc:288] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [CANDIDATE]: Term 1 election: Requested vote from peers a07fb89f73a54f18b01a637e58df4e17 (127.3.85.130:41895), ed73bf5e5c214e3998ac0dadfe48007c (127.3.85.129:33301)
I20260619 18:43:58.905676 16600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75106f93aeac484697a83eec69d563fd" candidate_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07fb89f73a54f18b01a637e58df4e17"
I20260619 18:43:58.905911 16600 raft_consensus.cc:3037] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:58.909210 16600 raft_consensus.cc:2454] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131ccbc73d5343c1a19faa80f6f8e72f in term 1.
I20260619 18:43:58.910044 16669 leader_election.cc:302] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 131ccbc73d5343c1a19faa80f6f8e72f, a07fb89f73a54f18b01a637e58df4e17; no voters:
I20260619 18:43:58.910755 16455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75106f93aeac484697a83eec69d563fd" candidate_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed73bf5e5c214e3998ac0dadfe48007c"
I20260619 18:43:58.910996 16455 raft_consensus.cc:3037] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:43:58.914166 16455 raft_consensus.cc:2454] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131ccbc73d5343c1a19faa80f6f8e72f in term 1.
I20260619 18:43:58.915128 16809 raft_consensus.cc:2781] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:43:58.930822 16809 raft_consensus.cc:686] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 1 LEADER]: Becoming Leader. State: Replica: 131ccbc73d5343c1a19faa80f6f8e72f, State: Running, Role: LEADER
I20260619 18:43:58.931496 16809 consensus_queue.cc:237] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } }
I20260619 18:43:58.937665 16302 catalog_manager.cc:5495] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f reported cstate change: term changed from 0 to 1, leader changed from <none> to 131ccbc73d5343c1a19faa80f6f8e72f (127.3.85.131). New cstate: current_term: 1 leader_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131ccbc73d5343c1a19faa80f6f8e72f" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 43531 } health_report { overall_health: HEALTHY } } }
W20260619 18:43:58.949708 16784 tablet.cc:2369] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:43:58.983867 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:58.977574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.3.85.131:43531)
0619 18:43:58.977776 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:58.977782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:58.977832 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:43:58.977949 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:58.977953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:58.978050 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:58.978208 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:43:58.978667 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.979701 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.981154 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:58.981746 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:58.981875 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:58.982486 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:43:58.983179 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:43:58.983340 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:58.983508 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:43:58.983599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:43:58.992990 16455 raft_consensus.cc:1261] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 1 FOLLOWER]: Refusing update from remote peer 131ccbc73d5343c1a19faa80f6f8e72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:58.993074 16600 raft_consensus.cc:1261] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 1 FOLLOWER]: Refusing update from remote peer 131ccbc73d5343c1a19faa80f6f8e72f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:43:58.994127 16811 consensus_queue.cc:1035] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07fb89f73a54f18b01a637e58df4e17" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:58.994546 16809 consensus_queue.cc:1035] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed73bf5e5c214e3998ac0dadfe48007c" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:43:59.007808 16821 mvcc.cc:205] Tried to move back new op lower bound from 7298640441303732224 to 7298640441068646400. Current Snapshot: MvccSnapshot[applied={T|T < 7298640441303732224}]
I20260619 18:43:59.026785 16823 mvcc.cc:205] Tried to move back new op lower bound from 7298640441303732224 to 7298640441068646400. Current Snapshot: MvccSnapshot[applied={T|T < 7298640441303732224}]
I20260619 18:43:59.036334 16302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:59.036648 16302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:43:59.041811 16302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03bc8feb8d7642f4a0256efe611ccf9d: Sending DeleteTablet for 3 replicas of tablet 75106f93aeac484697a83eec69d563fd
I20260619 18:43:59.043263 16714 tablet_service.cc:1514] Processing DeleteTablet for tablet 75106f93aeac484697a83eec69d563fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35490
I20260619 18:43:59.043321 16580 tablet_service.cc:1514] Processing DeleteTablet for tablet 75106f93aeac484697a83eec69d563fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:57616
I20260619 18:43:59.043473 16435 tablet_service.cc:1514] Processing DeleteTablet for tablet 75106f93aeac484697a83eec69d563fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:43:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39912
I20260619 18:43:59.044359 16830 tablet_replica.cc:330] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: stopping tablet replica
I20260619 18:43:59.045027 16830 raft_consensus.cc:2229] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:59.045723 16830 raft_consensus.cc:2258] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:59.047087 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16340
I20260619 18:43:59.048519 16830 ts_tablet_manager.cc:1905] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:59.058823 16830 ts_tablet_manager.cc:1918] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:59.059181 16830 log.cc:1196] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/75106f93aeac484697a83eec69d563fd
I20260619 18:43:59.060019 16830 ts_tablet_manager.cc:1939] T 75106f93aeac484697a83eec69d563fd P a07fb89f73a54f18b01a637e58df4e17: Deleting consensus metadata
I20260619 18:43:59.062307 16831 tablet_replica.cc:330] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: stopping tablet replica
I20260619 18:43:59.062403 16289 catalog_manager.cc:4841] TS a07fb89f73a54f18b01a637e58df4e17 (127.3.85.130:41895): tablet 75106f93aeac484697a83eec69d563fd (table test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]) successfully deleted
I20260619 18:43:59.064011 16831 raft_consensus.cc:2229] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:43:59.064733 16831 raft_consensus.cc:2258] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:59.067081 16832 tablet_replica.cc:330] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: stopping tablet replica
I20260619 18:43:59.067574 16832 raft_consensus.cc:2229] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:43:59.068126 16832 raft_consensus.cc:2258] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:43:59.070375 16831 ts_tablet_manager.cc:1905] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:59.077165 16831 ts_tablet_manager.cc:1918] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:59.077394 16831 log.cc:1196] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/75106f93aeac484697a83eec69d563fd
I20260619 18:43:59.078017 16831 ts_tablet_manager.cc:1939] T 75106f93aeac484697a83eec69d563fd P 131ccbc73d5343c1a19faa80f6f8e72f: Deleting consensus metadata
I20260619 18:43:59.080083 16287 catalog_manager.cc:4841] TS 131ccbc73d5343c1a19faa80f6f8e72f (127.3.85.131:43531): tablet 75106f93aeac484697a83eec69d563fd (table test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]) successfully deleted
I20260619 18:43:59.080677 16832 ts_tablet_manager.cc:1905] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:43:59.088860 16832 ts_tablet_manager.cc:1918] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:43:59.089174 16832 log.cc:1196] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/75106f93aeac484697a83eec69d563fd
I20260619 18:43:59.089854 16832 ts_tablet_manager.cc:1939] T 75106f93aeac484697a83eec69d563fd P ed73bf5e5c214e3998ac0dadfe48007c: Deleting consensus metadata
I20260619 18:43:59.092046 16289 catalog_manager.cc:4841] TS ed73bf5e5c214e3998ac0dadfe48007c (127.3.85.129:33301): tablet 75106f93aeac484697a83eec69d563fd (table test-table [id=967a51d7a7ee48b9a1fedf4ec017a26e]) successfully deleted
I20260619 18:43:59.092829 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:59.048412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.3.85.129:33301)
0619 18:43:59.048674 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:59.048681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:59.059236 (+ 10555us) server_negotiation.cc:407] Connection header received
0619 18:43:59.060389 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:59.060399 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:59.060510 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:59.060599 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:59.063786 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.064948 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.067012 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.067681 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.067820 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:59.082907 (+ 15087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:59.082922 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:59.082938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:59.082982 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:59.086967 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.089958 (+ 2991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.089973 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.089976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.090211 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.090791 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.090796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.090799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.091274 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:59.091606 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:59.092274 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 18:43:59.092559 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 18:43:59.094302 16495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:59.188433 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16340
I20260619 18:43:59.207448 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16504
I20260619 18:43:59.222975 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:59.208322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.3.85.130:41895)
0619 18:43:59.208931 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:59.208937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:59.208958 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:43:59.209038 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:59.209043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:59.209106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:59.209230 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:59.210798 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.211515 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.212908 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.213348 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.213504 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:59.216254 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:59.216270 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:59.216287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:59.216324 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:59.219573 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.220367 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.220376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.220378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.220526 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.220935 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.220941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.220944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.221336 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:59.221649 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:59.222073 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 18:43:59.222299 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260619 18:43:59.223738 16640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:59.330715 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16504
I20260619 18:43:59.349026 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16649
I20260619 18:43:59.365370 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:43:59.350208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.3.85.131:43531)
0619 18:43:59.350397 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:43:59.350401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:43:59.350417 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:43:59.350535 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:43:59.350539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:43:59.350612 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:43:59.350738 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:43:59.352116 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.352946 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.354421 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:43:59.354962 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:43:59.355141 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:43:59.358183 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:43:59.358196 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:43:59.358213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:43:59.358258 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:43:59.361989 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.363013 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.363031 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.363034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.363199 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:43:59.363633 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:43:59.363640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:43:59.363644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:43:59.364038 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:43:59.364368 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 18:43:59.364723 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:43:59.364951 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:43:59.366102 16774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:59.486282 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16649
I20260619 18:43:59.506657 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16270
I20260619 18:43:59.508458 16333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:43:59.626314 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16270
I20260619 18:43:59.649456 3414 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:43:59.996421 3414 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:43:59.996712 3414 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:44:00.343606 3414 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:44:00.343 UTC [15322] LOG: received smart shutdown request
2026-06-19 18:44:00.347 UTC [15322] LOG: background worker "logical replication launcher" (PID 15331) exited with exit code 1
2026-06-19 18:44:00.347 UTC [15326] LOG: shutting down
2026-06-19 18:44:00.549 UTC [15322] LOG: database system is shut down
2026-06-19T18:44:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57719 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:00 dist-test-slave-09fp krb5kdc[16846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:00 dist-test-slave-09fp krb5kdc[16846](info): set up 2 sockets
Jun 19 18:44:00 dist-test-slave-09fp krb5kdc[16846](info): commencing operation
krb5kdc: starting...
W20260619 18:44:01.819320 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:44:01 dist-test-slave-09fp krb5kdc[16846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:44:02.088589 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46111
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:02.444765 16859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:02.445111 16859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:02.445484 16859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:02.453557 16859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:02.453701 16859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:02.453799 16859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:02.453889 16859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:02.454002 16859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:02.464952 16859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:02.466557 16859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:02.467998 16859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:02.481676 16864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:02.481622 16865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:02.481590 16867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:02.483664 16859 server_base.cc:1034] running on GCE node
I20260619 18:44:02.484719 16859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:02.486529 16859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:02.487836 16859 hybrid_clock.cc:648] HybridClock initialized: now 1781894642487794 us; error 48 us; skew 500 ppm
I20260619 18:44:02.488268 16859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:02 dist-test-slave-09fp krb5kdc[16846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:02.496814 16859 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:02.498822 16859 webserver.cc:460] Webserver started at http://127.3.85.190:45953/ using document root <none> and password file <none>
I20260619 18:44:02.499657 16859 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:02.499832 16859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:02.500178 16859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:02.503595 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "7ef706b95d0c48548f38a51e4a5b4b0b"
format_stamp: "Formatted at 2026-06-19 18:44:02 on dist-test-slave-09fp"
server_key: "65f1953b9952af5a685cdf72e44fc0c6"
server_key_iv: "a728d095e499035f5f75eedf5cfc1b16"
server_key_version: "encryptionkey@0"
I20260619 18:44:02.504582 16859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "7ef706b95d0c48548f38a51e4a5b4b0b"
format_stamp: "Formatted at 2026-06-19 18:44:02 on dist-test-slave-09fp"
server_key: "65f1953b9952af5a685cdf72e44fc0c6"
server_key_iv: "a728d095e499035f5f75eedf5cfc1b16"
server_key_version: "encryptionkey@0"
I20260619 18:44:02.511487 16859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:44:02.516616 16874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:02.518911 16859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:02.519196 16859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "7ef706b95d0c48548f38a51e4a5b4b0b"
format_stamp: "Formatted at 2026-06-19 18:44:02 on dist-test-slave-09fp"
server_key: "65f1953b9952af5a685cdf72e44fc0c6"
server_key_iv: "a728d095e499035f5f75eedf5cfc1b16"
server_key_version: "encryptionkey@0"
I20260619 18:44:02.519465 16859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:02.533023 16859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:02.539053 16859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:02.539415 16859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:02.561266 16859 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46111
I20260619 18:44:02.561301 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46111 every 8 connection(s)
I20260619 18:44:02.562929 16859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:02.563903 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16859
I20260619 18:44:02.564136 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:44:02.564815 3414 external_mini_cluster.cc:1428] Setting key 4fdbbf11b37885704276f558ce65eaec
I20260619 18:44:02.567747 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:02.575162 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.85.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.189@KRBTEST.COM" created.
Jun 19 18:44:02 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:02.602495 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.578926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46111 (local address 127.0.0.1:51804)
0619 18:44:02.580230 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.580268 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:02.582353 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:02.583471 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:02.583479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:02.584253 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:02.584630 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:02.584649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:02.586435 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:02.586499 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:02.587691 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:02.587706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:02.588771 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:02.588778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:02.588945 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:02.589719 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:02.589743 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:02.592216 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:02.597469 (+ 5253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:02.597485 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:02.597502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:02.597958 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:02.598620 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:02.598623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:02.598626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:02.598795 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:02.599672 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:02.599686 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:02.599932 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 18:44:02.600206 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 18:44:02.600996 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":280,"thread_start_us":838,"threads_started":1}
I20260619 18:44:02.602519 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.578926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.3.85.190:46111)
0619 18:44:02.579897 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:02.579919 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:02.580425 (+ 506us) server_negotiation.cc:407] Connection header received
0619 18:44:02.582921 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:02.582928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:02.583212 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:02.583447 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:02.584865 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:02.586237 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:02.587883 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:02.588493 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:02.588734 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:02.592494 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:02.592503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:02.592517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:02.592551 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:02.597236 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:02.598146 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:02.598154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:02.598156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:02.598399 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:02.598924 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:02.598929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:02.598933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:02.599513 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:02.599764 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:02.600343 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 18:44:02.600868 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":698,"thread_start_us":460,"threads_started":1}
I20260619 18:44:02.605991 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:02.607604 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.606839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49212)
0619 18:44:02.607219 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.607397 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
W20260619 18:44:02.608037 16876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
I20260619 18:44:02.610059 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.609491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49214)
0619 18:44:02.609720 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.609832 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 18:44:02.610364 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:45977: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
I20260619 18:44:02.623771 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:45977
--webserver_interface=127.3.85.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:02.659842 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } attempt: 1
I20260619 18:44:02.661160 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.660594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49226)
0619 18:44:02.660846 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.660974 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 18:44:02.662475 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.661969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49228)
0619 18:44:02.662144 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.662237 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:44:02.662731 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:45977: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
I20260619 18:44:02.731001 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } attempt: 2
I20260619 18:44:02.732261 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.731734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49238)
0619 18:44:02.731972 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.732084 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:44:02.733489 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.733018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49250)
0619 18:44:02.733197 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.733303 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 18:44:02.733814 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:45977: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
I20260619 18:44:02.825150 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } attempt: 3
I20260619 18:44:02.826658 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.825918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49260)
0619 18:44:02.826207 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.826369 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 18:44:02.828101 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.827472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49270)
0619 18:44:02.827772 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.827871 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260619 18:44:02.828372 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:45977: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
I20260619 18:44:02.971712 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } attempt: 4
I20260619 18:44:02.973138 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.972476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49286)
0619 18:44:02.972839 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.972959 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 18:44:02.974572 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:02.974029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49290)
0619 18:44:02.974227 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:02.974354 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 18:44:02.974841 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:45977: Network error: Client connection negotiation failed: client connection to 127.3.85.189:45977: connect: Connection refused (error 111)
W20260619 18:44:02.988081 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:02.988483 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:02.988662 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:02.996663 16933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:02.996874 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:02.997032 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:02.997138 16933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:02.997200 16933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:03.007731 16933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.85.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:03.009413 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:03.010949 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:03.024806 16939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:03.024835 16941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:03.024744 16938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:03.025439 16933 server_base.cc:1034] running on GCE node
I20260619 18:44:03.026382 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:03.028162 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:03.029445 16933 hybrid_clock.cc:648] HybridClock initialized: now 1781894643029395 us; error 49 us; skew 500 ppm
I20260619 18:44:03.029913 16933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:03.033908 16933 init.cc:377] Logged in from keytab as kudu/127.3.85.189@KRBTEST.COM (short username kudu)
I20260619 18:44:03.035845 16933 webserver.cc:460] Webserver started at http://127.3.85.189:45455/ using document root <none> and password file <none>
I20260619 18:44:03.036609 16933 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:03.036754 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:03.037035 16933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:03.040544 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data/instance:
uuid: "57ab59583257447a917459ad315bbd7e"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "53e35bfb6bb056c029cf4f18ad2920d6"
server_key_iv: "958fa994a61989428b1ab433f697aa56"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.041718 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal/instance:
uuid: "57ab59583257447a917459ad315bbd7e"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "53e35bfb6bb056c029cf4f18ad2920d6"
server_key_iv: "958fa994a61989428b1ab433f697aa56"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.048278 16933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:44:03.052517 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:03.054315 16933 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:44:03.054621 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "57ab59583257447a917459ad315bbd7e"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "53e35bfb6bb056c029cf4f18ad2920d6"
server_key_iv: "958fa994a61989428b1ab433f697aa56"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.054955 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:03.071058 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:03.075953 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:03.076222 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:03.097680 16933 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.189:45977
I20260619 18:44:03.097708 16999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.189:45977 every 8 connection(s)
I20260619 18:44:03.099383 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
I20260619 18:44:03.103380 17000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:03.109786 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 16933
I20260619 18:44:03.110016 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-1/wal/instance
I20260619 18:44:03.110662 3414 external_mini_cluster.cc:1428] Setting key 79c971d1419a7cea03e5653287030afc
I20260619 18:44:03.111089 17000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.188@KRBTEST.COM" created.
I20260619 18:44:03.138345 17002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.115029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.3.85.190:46111)
0619 18:44:03.115651 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.115661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.116803 (+ 1142us) server_negotiation.cc:407] Connection header received
0619 18:44:03.119862 (+ 3059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.119870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.119970 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.120116 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.121924 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.122862 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.124708 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.125169 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.125331 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.129765 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.129775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.129780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.129846 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.134167 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.135301 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.135306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.135308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.135448 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.135930 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.135934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.135936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.136248 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.136371 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.137548 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.137907 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
I20260619 18:44:03.139818 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.115674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46111 (local address 127.0.0.1:51806)
0619 18:44:03.116569 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.116632 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.119390 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.120397 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.120406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.120998 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.121694 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.121715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.123065 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.123070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.124509 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.124531 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.125447 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.125455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.125726 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.126591 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.126632 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.129378 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.134431 (+ 5053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.134440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.134540 (+ 100us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.135133 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.135593 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.135600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.135606 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.135823 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.136425 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.136432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.137134 (+ 702us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.137406 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.138035 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":355,"threads_started":1}
I20260619 18:44:03.141978 17000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:44:03.158232 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.142797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49304)
0619 18:44:03.142961 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.142977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.143074 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.144161 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.144165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.144648 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.144790 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.144799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.146219 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.146226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.147177 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.147190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.148121 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.148127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.148332 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.149047 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.149066 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.151609 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.155436 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.155444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.155448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.155865 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.156317 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.156323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.156326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.156487 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.157168 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.157175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.157338 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.157489 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.157840 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:44:03.158394 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.143105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.3.85.189:45977)
0619 18:44:03.143691 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.143697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.143723 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:44:03.143774 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.143779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.143973 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.144160 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.144963 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.146045 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.147340 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.147934 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.148088 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.151894 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.151912 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.151928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.151961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.155243 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.156044 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.156058 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.156060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.156187 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.156626 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.156633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.156636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.157021 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.157275 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.157657 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.158001 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260619 18:44:03.161303 17000 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:03.162801 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.162206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41152)
0619 18:44:03.162418 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.162628 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 18:44:03.163221 16949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.165232 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.164844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41160)
0619 18:44:03.165005 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.165099 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:44:03.165761 17000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
I20260619 18:44:03.170985 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:36873
--webserver_interface=127.3.85.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:03.215207 17000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 1
I20260619 18:44:03.216565 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.216055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41162)
0619 18:44:03.216290 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.216403 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:44:03.217983 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.217444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41172)
0619 18:44:03.217679 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.217788 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 18:44:03.218333 17000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.274209 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } attempt: 5
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:44:03.286697 17000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 2
I20260619 18:44:03.287964 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.287410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41176)
0619 18:44:03.287645 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.287755 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:44:03.289278 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.288843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41186)
0619 18:44:03.289031 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.289121 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 18:44:03.289608 17000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.293435 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.275323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49308)
0619 18:44:03.275587 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.275604 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.275727 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.276311 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.276324 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.276816 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.277000 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.277010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.278579 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.278583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.279494 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.279508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.280791 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.280796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.280947 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.282016 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.282040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.284501 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.290104 (+ 5603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.290110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.290113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.290658 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.291260 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.291264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.291267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.291424 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.292321 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.292325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.292598 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.292792 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.293037 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 18:44:03.293579 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.275323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.3.85.189:45977)
0619 18:44:03.275645 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.275651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.275676 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:44:03.275883 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.275889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.275975 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.276095 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.277197 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.278327 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.279704 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.280546 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.280714 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.284796 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.284803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.284808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.284858 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.289882 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.290806 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.290813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.290816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.290982 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.291634 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.291640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.291643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.292147 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.292345 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.292969 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.293240 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260619 18:44:03.294716 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:03.296000 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.295522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41188)
0619 18:44:03.295757 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.295853 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 18:44:03.297055 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.296707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41190)
0619 18:44:03.296868 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.296953 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:44:03.297293 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.348567 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 1
I20260619 18:44:03.349987 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.349357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41202)
0619 18:44:03.349674 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.349812 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 18:44:03.351222 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.350820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41208)
0619 18:44:03.350993 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.351096 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:44:03.351471 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.380944 17000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 3
I20260619 18:44:03.382396 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.381757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41214)
0619 18:44:03.382008 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.382120 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:44:03.383781 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.383306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41218)
0619 18:44:03.383545 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.383647 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260619 18:44:03.384001 17000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.419754 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 2
I20260619 18:44:03.421131 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.420546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41230)
0619 18:44:03.420796 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.420920 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 18:44:03.422432 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.421929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41238)
0619 18:44:03.422143 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.422268 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:44:03.422751 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.527254 17000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 4
I20260619 18:44:03.528702 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.527988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41242)
0619 18:44:03.528298 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.528426 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 18:44:03.529029 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 3
I20260619 18:44:03.530272 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.529691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41256)
0619 18:44:03.529916 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.530030 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 18:44:03.530635 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.530102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41264)
0619 18:44:03.530312 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.530430 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:44:03.530925 17000 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
I20260619 18:44:03.531523 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.531085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41270)
0619 18:44:03.531308 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.531395 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 18:44:03.531798 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:36873: Network error: Client connection negotiation failed: client connection to 127.3.85.188:36873: connect: Connection refused (error 111)
W20260619 18:44:03.544942 17008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:03.545341 17008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:03.545506 17008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:03.553915 17008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:03.554119 17008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:03.554242 17008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:03.554333 17008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:03.554418 17008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:03.568401 17008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46111,127.3.85.189:45977,127.3.85.188:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.85.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:03.570159 17008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:03.571995 17008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:03.586403 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:03.586628 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:03.586621 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:03.587440 17008 server_base.cc:1034] running on GCE node
I20260619 18:44:03.588462 17008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:03.590171 17008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:03.591496 17008 hybrid_clock.cc:648] HybridClock initialized: now 1781894643591468 us; error 65 us; skew 500 ppm
I20260619 18:44:03.591969 17008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:03.595621 17008 init.cc:377] Logged in from keytab as kudu/127.3.85.188@KRBTEST.COM (short username kudu)
I20260619 18:44:03.597494 17008 webserver.cc:460] Webserver started at http://127.3.85.188:33065/ using document root <none> and password file <none>
I20260619 18:44:03.598289 17008 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:03.598518 17008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:03.598868 17008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:03.602108 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data/instance:
uuid: "12b828414678429ea021a07bccb9119b"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "39e336dbfcab1f73c5e32896e1cbe86f"
server_key_iv: "de237190517d5d32fc21089d3f00ff74"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.603081 17008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal/instance:
uuid: "12b828414678429ea021a07bccb9119b"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "39e336dbfcab1f73c5e32896e1cbe86f"
server_key_iv: "de237190517d5d32fc21089d3f00ff74"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.609113 17008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:44:03.613286 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:03.614991 17008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:03.615228 17008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "12b828414678429ea021a07bccb9119b"
format_stamp: "Formatted at 2026-06-19 18:44:03 on dist-test-slave-09fp"
server_key: "39e336dbfcab1f73c5e32896e1cbe86f"
server_key_iv: "de237190517d5d32fc21089d3f00ff74"
server_key_version: "encryptionkey@0"
I20260619 18:44:03.615465 17008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:03.632364 17008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:03.637269 17008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:03.637537 17008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:03.658896 17008 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.188:36873
I20260619 18:44:03.658998 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.188:36873 every 8 connection(s)
I20260619 18:44:03.660597 17008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
I20260619 18:44:03.664916 17075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:03.667219 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17008
I20260619 18:44:03.667384 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894429649633-3414-0/minicluster-data/master-2/wal/instance
I20260619 18:44:03.667974 3414 external_mini_cluster.cc:1428] Setting key 13c91cf1d6813559efc902bccbe1c245
I20260619 18:44:03.674785 17075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:03.696791 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.670191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.3.85.188:36873)
0619 18:44:03.671246 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.671263 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.671327 (+ 64us) server_negotiation.cc:407] Connection header received
0619 18:44:03.674036 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.674082 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.674691 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.675237 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.676943 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.678957 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.680810 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.681807 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.682027 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.687528 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.687540 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.687561 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.687605 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.691239 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.692269 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.692283 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.692296 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.692554 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.693103 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.693107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.693110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.693796 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.694203 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.694686 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.695241 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":481,"threads_started":1}
I20260619 18:44:03.697563 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.678356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46111 (local address 127.0.0.1:51810)
0619 18:44:03.679571 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.679621 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.679804 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.680256 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.680262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.680892 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.681213 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.681223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.682674 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.682679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.684470 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.684496 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.685855 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.685869 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.686126 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.686967 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.686994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.689999 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.694041 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.694054 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.694063 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.694777 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.695367 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.695372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.695375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.695583 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.696266 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.696274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.696483 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.696724 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.697098 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":748,"threads_started":1}
I20260619 18:44:03.698233 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.678265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.3.85.190:46111)
0619 18:44:03.678898 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.678908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.679750 (+ 842us) server_negotiation.cc:407] Connection header received
0619 18:44:03.679953 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.679960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.680057 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.680177 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.681409 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.682416 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.684703 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.685429 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.685600 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.690290 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.690296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.690300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.690349 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.693821 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.694965 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.694972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.694976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.695173 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.695782 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.695786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.695788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.696106 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.696237 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.696846 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.697918 (+ 1072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260619 18:44:03.699700 17075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:44:03.709048 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 4
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:03.717164 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.700756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.3.85.189:45977)
0619 18:44:03.700996 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.701002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.701023 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:44:03.701126 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.701135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.701220 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.701320 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.702215 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.703361 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.705025 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.705751 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.705927 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.709962 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.709970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.709974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.710020 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.713697 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.714624 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.714634 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.714637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.714781 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.715363 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.715370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.715373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.715660 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.715839 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.716481 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.716751 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:44:03.717281 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.700524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:45977 (local address 127.0.0.1:49316)
0619 18:44:03.700862 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.700882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.701000 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.701374 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.701379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.701876 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.702028 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.702037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.703518 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.703524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.704857 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.704869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.705978 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.705984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.706221 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.706895 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.706920 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.709649 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.713853 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.713859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.713863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.714427 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.714955 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.714958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.714960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.715123 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.715798 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.715803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.716191 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.716366 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.716934 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894642, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:44:03.719190 17075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:44:03.724149 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.700786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.3.85.190:46111)
0619 18:44:03.700959 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.700964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.701201 (+ 237us) server_negotiation.cc:407] Connection header received
0619 18:44:03.701378 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.701383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.701471 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.701582 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.703341 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.704502 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.706378 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.707085 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.707227 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.712596 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.712606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.712610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.712645 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.718369 (+ 5724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.719563 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.719570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.719573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.719773 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.720269 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.720275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.720278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.721642 (+ 1364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.722017 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.723459 (+ 1442us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.723738 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:44:03.729044 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.709772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41292)
0619 18:44:03.710007 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.710028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.710144 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.711162 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.711168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.711655 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.711893 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.711910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.713317 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.713324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.714720 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.714734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.715911 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.715917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.716166 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.716726 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.716750 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.720218 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.725379 (+ 5161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.725385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.725390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.726168 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.726919 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.726925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.726929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.727126 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.728021 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.728028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.728247 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.728400 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.728770 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:44:03.729982 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.710213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.3.85.188:36873)
0619 18:44:03.710412 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.710418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.710436 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:44:03.710526 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.710531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.710626 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.710735 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.712053 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.713154 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.714870 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.715701 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.716477 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.720417 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.720424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.720429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.720470 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.725205 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.726379 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.726385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.726389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.726754 (+ 365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.727303 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.727310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.727316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.727839 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.729358 (+ 1519us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.729497 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.729745 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:44:03.737255 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: Bootstrap starting.
I20260619 18:44:03.740146 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.720218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41304)
0619 18:44:03.720634 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.720655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.720833 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.722001 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.722024 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.722585 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.722757 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.722766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.724898 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.724904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.726535 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.726551 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.727495 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.727506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.727804 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.728468 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.728486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.731788 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.736415 (+ 4627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.736421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.736425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.736956 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.737504 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.737511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.737515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.737707 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.738559 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.738565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.738761 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.738923 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.739654 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":56}
I20260619 18:44:03.740469 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.720586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.3.85.188:36873)
0619 18:44:03.721658 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.721665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.721689 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:44:03.721745 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.721752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.721833 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.721987 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.722965 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.724725 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.726812 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.727326 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.727486 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.732053 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.732061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.732066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.732109 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.736218 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.737128 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.737133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.737136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.737334 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.737890 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.737896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.737900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.738351 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.738559 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.739108 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.739973 (+ 865us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":511,"server-negotiator.queue_time_us":921,"thread_start_us":651,"threads_started":1}
I20260619 18:44:03.742059 16926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:03.743399 16926 log.cc:824] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:03.746565 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: Bootstrap starting.
I20260619 18:44:03.746981 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.726347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.3.85.189:45977)
0619 18:44:03.726586 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.726592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.726613 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:44:03.726679 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.726686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.726771 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.726887 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.728732 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.730865 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.732429 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.733262 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.733450 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.738557 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.738565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.738570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.738618 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.742782 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.743852 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.743858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.743861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.744051 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.744579 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.744585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.744589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.744959 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.745187 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.746315 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.746631 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:44:03.747010 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: No bootstrap required, opened a new log
I20260619 18:44:03.751025 17075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:03.752434 17075 log.cc:824] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:03.753690 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:44:03.756708 17075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: No bootstrap required, opened a new log
I20260619 18:44:03.757563 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.758069 16926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:03.758221 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef706b95d0c48548f38a51e4a5b4b0b, State: Initialized, Role: FOLLOWER
I20260619 18:44:03.759078 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.760250 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.760787 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.762051 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:03.769765 17088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:03.770932 17088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.771706 17075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.772249 17075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:03.772418 17075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b828414678429ea021a07bccb9119b, State: Initialized, Role: FOLLOWER
I20260619 18:44:03.773221 17075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.781096 17088 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57ab59583257447a917459ad315bbd7e (127.3.85.189:45977), 12b828414678429ea021a07bccb9119b (127.3.85.188:36873)
W20260619 18:44:03.793161 16880 tablet.cc:2369] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:03.798065 17075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:03.800034 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.760025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.3.85.190:46111)
0619 18:44:03.760210 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.760219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.770142 (+ 9923us) server_negotiation.cc:407] Connection header received
0619 18:44:03.771234 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.771241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.771325 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.771419 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.774588 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.776241 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.779807 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.780835 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.781011 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.786617 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.786628 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.786636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.786702 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.793720 (+ 7018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.796191 (+ 2471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.796207 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.796211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.796539 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.797485 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.797492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.797495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.798231 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.798602 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.799306 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.799599 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:44:03.798125 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b828414678429ea021a07bccb9119b" is_pre_election: true
I20260619 18:44:03.802341 17050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef706b95d0c48548f38a51e4a5b4b0b in term 0.
I20260619 18:44:03.803400 16875 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b828414678429ea021a07bccb9119b, 7ef706b95d0c48548f38a51e4a5b4b0b; no voters:
I20260619 18:44:03.804276 17088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:03.804513 17088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:03.804646 17088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:03.809729 17088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.814677 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12b828414678429ea021a07bccb9119b"
I20260619 18:44:03.815248 17050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:03.819800 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.763655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.3.85.188:36873)
0619 18:44:03.763939 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.763948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.774794 (+ 10846us) server_negotiation.cc:407] Connection header received
0619 18:44:03.775991 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.775998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.776098 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.776176 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.782804 (+ 6628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.784056 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.789554 (+ 5498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.790373 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.790511 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.806329 (+ 15818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.806341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.806349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.806419 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.811289 (+ 4870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.812621 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.812630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.812635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.813178 (+ 543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.814394 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.814403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.814409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.815612 (+ 1203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.816303 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.817903 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.818687 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:44:03.823344 17050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef706b95d0c48548f38a51e4a5b4b0b in term 1.
I20260619 18:44:03.824970 16875 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12b828414678429ea021a07bccb9119b, 7ef706b95d0c48548f38a51e4a5b4b0b; no voters:
I20260619 18:44:03.797847 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.825601 17088 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [CANDIDATE]: Term 1 election: Requested vote from peers 57ab59583257447a917459ad315bbd7e (127.3.85.189:45977), 12b828414678429ea021a07bccb9119b (127.3.85.188:36873)
I20260619 18:44:03.825893 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.826267 17088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:03.826742 17088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [term 1 LEADER]: Becoming Leader. State: Replica: 7ef706b95d0c48548f38a51e4a5b4b0b, State: Running, Role: LEADER
I20260619 18:44:03.829711 17088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.830194 17000 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } attempt: 5
Jun 19 18:44:03 dist-test-slave-09fp krb5kdc[16846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894643, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:44:03.864413 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.830912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:36873 (local address 127.0.0.1:41322)
0619 18:44:03.831184 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:03.831210 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:03.831381 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:03.832082 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:03.832090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:03.832837 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:03.833137 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.833154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.835430 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.835439 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.838123 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:03.838167 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.840252 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.840263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:03.840571 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.842891 (+ 2320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:03.842920 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:03.847110 (+ 4190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:03.858618 (+ 11508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.858627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.858634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.859414 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.860674 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:03.860682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:03.860689 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:03.860951 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:03.862786 (+ 1835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:03.862796 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:03.863044 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 18:44:03.863291 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 18:44:03.863952 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 18:44:03.865994 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.830960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.3.85.188:36873)
0619 18:44:03.831668 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.831678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.831702 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:44:03.831773 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.831782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.831884 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.833363 (+ 1479us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.833569 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.835164 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.838528 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.839521 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.841069 (+ 1548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.847457 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.847467 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.847474 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.847537 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.858180 (+ 10643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.859713 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.859721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.859771 (+ 50us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.860365 (+ 594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.861166 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.861176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.861183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.862508 (+ 1325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.863066 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.863811 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.864655 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543}
I20260619 18:44:03.867458 17100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef706b95d0c48548f38a51e4a5b4b0b. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.867923 17100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:03.870342 17109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:03.871809 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:03.770859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.3.85.189:45977)
0619 18:44:03.771049 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:03.771054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:03.790218 (+ 19164us) server_negotiation.cc:407] Connection header received
0619 18:44:03.790890 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:03.790899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:03.790995 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:03.791072 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:03.799124 (+ 8052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.800866 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.841433 (+ 40567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:03.842394 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:03.842588 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:03.854302 (+ 11714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:03.854310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:03.854318 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:03.854378 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:03.860082 (+ 5704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.868715 (+ 8633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.868722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.868726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.868966 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:03.869526 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:03.869532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:03.869536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:03.869984 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:03.870297 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:03.871139 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 18:44:03.871435 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:44:03.873028 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: Bootstrap starting.
I20260619 18:44:03.877792 17000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:03.879648 17000 log.cc:824] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:03.884604 17000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: No bootstrap required, opened a new log
W20260619 18:44:03.889859 17029 tablet.cc:2369] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:03.893790 17109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:03.907975 17119 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:44:03.907533 17000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.908329 17000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:03.908730 17000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ab59583257447a917459ad315bbd7e, State: Initialized, Role: FOLLOWER
I20260619 18:44:03.925321 17000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } }
I20260619 18:44:03.925630 17050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [term 1 FOLLOWER]: Refusing update from remote peer 7ef706b95d0c48548f38a51e4a5b4b0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:03.926795 17088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:03.928247 17121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.929203 17121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.916018 17008 master_runner.cc:186] Error getting master registration for 127.3.85.189:45977: OK, instance_id { permanent_uuid: "57ab59583257447a917459ad315bbd7e" instance_seqno: 1781894643091442 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:44:03.930122 17008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:44:03.912278 16859 master_runner.cc:186] Error getting master registration for 127.3.85.189:45977: OK, instance_id { permanent_uuid: "57ab59583257447a917459ad315bbd7e" instance_seqno: 1781894643091442 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:44:03.936558 16859 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:44:03.938521 17000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:03.938953 16973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:03.940630 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ab59583257447a917459ad315bbd7e" is_pre_election: true
I20260619 18:44:03.942509 16974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ab59583257447a917459ad315bbd7e"
I20260619 18:44:03.944378 16973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [term 1 FOLLOWER]: Refusing update from remote peer 7ef706b95d0c48548f38a51e4a5b4b0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:03.945533 17088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:03.952947 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef706b95d0c48548f38a51e4a5b4b0b. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.953274 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.957206 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.957599 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:03.971438 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.971771 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.974120 17109 catalog_manager.cc:1347] Generated new cluster ID: 87b9e95a1a3f4d33b287be514c26a656
I20260619 18:44:03.974442 17109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:03.976784 17121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef706b95d0c48548f38a51e4a5b4b0b. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.977149 17121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:03.978611 17100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:03.978943 17100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:03.980813 17127 mvcc.cc:205] Tried to move back new op lower bound from 7298640461502087168 to 7298640461214920704. Current Snapshot: MvccSnapshot[applied={T|T < 7298640461502087168}]
I20260619 18:44:04.000001 17121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef706b95d0c48548f38a51e4a5b4b0b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46111 } } peers { permanent_uuid: "57ab59583257447a917459ad315bbd7e" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 45977 } } peers { permanent_uuid: "12b828414678429ea021a07bccb9119b" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 36873 } } }
I20260619 18:44:04.000355 17121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:44:04.029999 17139 catalog_manager.cc:1259] Loaded cluster ID: 87b9e95a1a3f4d33b287be514c26a656
I20260619 18:44:04.030223 17139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: loading cluster ID for follower catalog manager: success
W20260619 18:44:04.031968 17139 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260619 18:44:04.039753 17139 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:44:04.040215 17109 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:04.042678 17109 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:04.063850 17109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef706b95d0c48548f38a51e4a5b4b0b: Generated new TSK 0
I20260619 18:44:04.065220 17109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:44:04.914011 17119 catalog_manager.cc:1259] Loaded cluster ID: 87b9e95a1a3f4d33b287be514c26a656
I20260619 18:44:04.914155 17119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: loading cluster ID for follower catalog manager: success
I20260619 18:44:04.918154 17119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: acquiring CA information for follower catalog manager: success
I20260619 18:44:04.921478 17119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12b828414678429ea021a07bccb9119b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:44:05.043893 17139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: acquiring CA information for follower catalog manager: success
I20260619 18:44:05.046952 17139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57ab59583257447a917459ad315bbd7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:44:05.163657 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16859
I20260619 18:44:05.164865 16921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:05.280601 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16859
W20260619 18:44:05.302743 17027 connection.cc:535] server connection from 127.0.0.1:41292 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:44:05.303131 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 16933
I20260619 18:44:05.304320 16995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:05.421629 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 16933
I20260619 18:44:05.443805 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17008
I20260619 18:44:05.445060 17070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:05.555979 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17008
2026-06-19T18:44:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4867 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:05 dist-test-slave-09fp krb5kdc[17151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:05 dist-test-slave-09fp krb5kdc[17151](info): set up 2 sockets
Jun 19 18:44:05 dist-test-slave-09fp krb5kdc[17151](info): commencing operation
krb5kdc: starting...
W20260619 18:44:06.671135 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:44:06 dist-test-slave-09fp krb5kdc[17151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:44:06.928445 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:36737
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:36737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:07.276857 17164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:07.277272 17164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:07.277432 17164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:07.285176 17164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:07.285362 17164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:07.285485 17164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:07.285595 17164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:07.285673 17164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:07.296403 17164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:36737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:07.297904 17164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:07.299357 17164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:07.313186 17169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:07.313148 17172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:07.313120 17170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:07.313732 17164 server_base.cc:1034] running on GCE node
I20260619 18:44:07.314786 17164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:07.316628 17164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:07.317904 17164 hybrid_clock.cc:648] HybridClock initialized: now 1781894647317852 us; error 40 us; skew 500 ppm
I20260619 18:44:07.318341 17164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:07 dist-test-slave-09fp krb5kdc[17151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:07.322186 17164 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:07.324138 17164 webserver.cc:460] Webserver started at http://127.3.85.190:39577/ using document root <none> and password file <none>
I20260619 18:44:07.324944 17164 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:07.325110 17164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:07.325462 17164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:07.328918 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "4390fc099b6d41b1bac3b87b96d0586b"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-09fp"
server_key: "6c71a43a79720bfca61a09ba87d1b1c6"
server_key_iv: "4ea36e3bdf7d5c63789e1c8a60cbe7bd"
server_key_version: "encryptionkey@0"
I20260619 18:44:07.329819 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "4390fc099b6d41b1bac3b87b96d0586b"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-09fp"
server_key: "6c71a43a79720bfca61a09ba87d1b1c6"
server_key_iv: "4ea36e3bdf7d5c63789e1c8a60cbe7bd"
server_key_version: "encryptionkey@0"
I20260619 18:44:07.336093 17164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:44:07.340029 17179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:07.341579 17164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:44:07.341841 17164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "4390fc099b6d41b1bac3b87b96d0586b"
format_stamp: "Formatted at 2026-06-19 18:44:07 on dist-test-slave-09fp"
server_key: "6c71a43a79720bfca61a09ba87d1b1c6"
server_key_iv: "4ea36e3bdf7d5c63789e1c8a60cbe7bd"
server_key_version: "encryptionkey@0"
I20260619 18:44:07.342096 17164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:07.364547 17164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:07.369520 17164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:07.369858 17164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:07.390040 17164 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:36737
I20260619 18:44:07.390084 17230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:36737 every 8 connection(s)
I20260619 18:44:07.391649 17164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:07.394526 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17164
I20260619 18:44:07.394783 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:44:07.395612 3414 external_mini_cluster.cc:1428] Setting key 465b8e10535821d68c302390adfb9bec
I20260619 18:44:07.396521 17231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:44:07 dist-test-slave-09fp krb5kdc[17151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:07.412626 17231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b: Bootstrap starting.
I20260619 18:44:07.416190 17231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:07.417327 17231 log.cc:824] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:07.420559 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:07.397678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.3.85.190:36737)
0619 18:44:07.398598 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:07.398611 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:07.398653 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:44:07.400268 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:07.400292 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:07.400732 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:07.401106 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:07.403113 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:07.404529 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:07.406586 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:07.407358 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:07.407519 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:07.412653 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:07.412662 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:07.412680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:07.412730 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:07.416025 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:07.416983 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:07.416991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:07.416998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:07.417194 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:07.417615 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:07.417620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:07.417626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:07.418203 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:07.418578 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:07.419068 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 18:44:07.419535 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":417,"threads_started":1}
I20260619 18:44:07.420658 17231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b: No bootstrap required, opened a new log
I20260619 18:44:07.424676 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:44:07.429155 17231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } }
I20260619 18:44:07.429626 17231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:07.429742 17231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4390fc099b6d41b1bac3b87b96d0586b, State: Initialized, Role: FOLLOWER
I20260619 18:44:07.430361 17231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } }
I20260619 18:44:07.430744 17231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:07.430902 17231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:07.431123 17231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:07.434342 17231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } }
I20260619 18:44:07.435014 17231 leader_election.cc:302] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4390fc099b6d41b1bac3b87b96d0586b; no voters:
I20260619 18:44:07.435686 17231 leader_election.cc:288] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:44:07.435890 17236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:07.436689 17236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [term 1 LEADER]: Becoming Leader. State: Replica: 4390fc099b6d41b1bac3b87b96d0586b, State: Running, Role: LEADER
I20260619 18:44:07.437292 17236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } }
I20260619 18:44:07.437963 17231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:07.440428 17238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4390fc099b6d41b1bac3b87b96d0586b. Latest consensus state: current_term: 1 leader_uuid: "4390fc099b6d41b1bac3b87b96d0586b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } } }
I20260619 18:44:07.440402 17237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4390fc099b6d41b1bac3b87b96d0586b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390fc099b6d41b1bac3b87b96d0586b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36737 } } }
I20260619 18:44:07.440932 17238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:07.441118 17237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:07.442911 17243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:07.448516 17243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:07.458949 17243 catalog_manager.cc:1347] Generated new cluster ID: cb80d75d78e14290820a047bc4895305
I20260619 18:44:07.459089 17243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:07.483855 17243 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:07.485847 17243 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:07.496872 17243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4390fc099b6d41b1bac3b87b96d0586b: Generated new TSK 0
I20260619 18:44:07.498059 17243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:44:07.537653 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17164
I20260619 18:44:07.539719 17226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:07.639596 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17164
2026-06-19T18:44:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2078 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:07 dist-test-slave-09fp krb5kdc[17259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:07 dist-test-slave-09fp krb5kdc[17259](info): set up 2 sockets
Jun 19 18:44:07 dist-test-slave-09fp krb5kdc[17259](info): commencing operation
krb5kdc: starting...
W20260619 18:44:08.759001 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:44:08 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:44:09.016140 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:09.360785 17272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:09.361119 17272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:09.361279 17272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:09.369050 17272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:09.369240 17272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:09.369388 17272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:09.369490 17272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:09.369561 17272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:09.380532 17272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:09.382004 17272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:09.383428 17272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:09.396052 17277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:09.396096 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:09.396011 17278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:09.397330 17272 server_base.cc:1034] running on GCE node
I20260619 18:44:09.398336 17272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:09.400192 17272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:09.401427 17272 hybrid_clock.cc:648] HybridClock initialized: now 1781894649401379 us; error 43 us; skew 500 ppm
I20260619 18:44:09.401882 17272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:09 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:09.405651 17272 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:09.407678 17272 webserver.cc:460] Webserver started at http://127.3.85.190:42733/ using document root <none> and password file <none>
I20260619 18:44:09.408540 17272 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:09.408821 17272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:09.409339 17272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:09.412847 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "afc045bcdf93480187bb2054ac9652c1"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-09fp"
server_key: "df9cd311e869518cb50beeac966f36db"
server_key_iv: "d13e5887a836c3ad0418151ff747f709"
server_key_version: "encryptionkey@0"
I20260619 18:44:09.413794 17272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "afc045bcdf93480187bb2054ac9652c1"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-09fp"
server_key: "df9cd311e869518cb50beeac966f36db"
server_key_iv: "d13e5887a836c3ad0418151ff747f709"
server_key_version: "encryptionkey@0"
I20260619 18:44:09.420255 17272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:44:09.425329 17287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:09.427795 17272 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 18:44:09.428076 17272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "afc045bcdf93480187bb2054ac9652c1"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-09fp"
server_key: "df9cd311e869518cb50beeac966f36db"
server_key_iv: "d13e5887a836c3ad0418151ff747f709"
server_key_version: "encryptionkey@0"
I20260619 18:44:09.428329 17272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:09.444650 17272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:09.449405 17272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:09.449750 17272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:09.469656 17272 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46203
I20260619 18:44:09.469687 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46203 every 8 connection(s)
I20260619 18:44:09.471338 17272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:09.475322 17339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:09.481415 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17272
I20260619 18:44:09.481673 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:44:09.482542 3414 external_mini_cluster.cc:1428] Setting key f5b6f93bc2437ba69f21c486bc451cf1
I20260619 18:44:09.488252 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap starting.
I20260619 18:44:09.492539 17339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:09.494028 17339 log.cc:824] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:09.498371 17339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: No bootstrap required, opened a new log
Jun 19 18:44:09 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:09.507743 17339 raft_consensus.cc:348] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:09.508249 17339 raft_consensus.cc:374] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:09.508443 17339 raft_consensus.cc:729] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Initialized, Role: FOLLOWER
I20260619 18:44:09.509166 17339 consensus_queue.cc:260] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:09.509449 17339 raft_consensus.cc:388] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:09.509614 17339 raft_consensus.cc:482] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:09.509809 17339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:09.509994 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:09.484909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.3.85.190:46203)
0619 18:44:09.485858 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:09.485872 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:09.485933 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:44:09.487880 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:09.487908 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:09.488447 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:09.488842 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:09.490929 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.492325 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.494384 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:09.495713 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:09.495889 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:09.501127 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:09.501136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:09.501158 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:09.501203 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:09.504796 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.505719 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.505727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.505739 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.505965 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:09.506418 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:09.506426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:09.506434 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:09.507180 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:09.507544 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:09.508041 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:44:09.508590 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":452,"threads_started":1}
I20260619 18:44:09.512593 17339 raft_consensus.cc:504] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:09.513296 17339 leader_election.cc:302] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afc045bcdf93480187bb2054ac9652c1; no voters:
I20260619 18:44:09.514047 17339 leader_election.cc:288] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:44:09.514216 17344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:09.515053 17344 raft_consensus.cc:686] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 LEADER]: Becoming Leader. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Running, Role: LEADER
I20260619 18:44:09.515676 17344 consensus_queue.cc:237] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:09.516479 17339 sys_catalog.cc:564] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:09.518874 17346 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afc045bcdf93480187bb2054ac9652c1. Latest consensus state: current_term: 1 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:09.518884 17345 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:09.519460 17345 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:09.519461 17346 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:09.520692 17352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:09.531626 17352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:09.545439 17352 catalog_manager.cc:1347] Generated new cluster ID: 2589391bb5984c47957ac33edf1bdbad
I20260619 18:44:09.545634 17352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:09.577929 17352 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:09.579524 17352 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:09.589030 17352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Generated new TSK 0
I20260619 18:44:09.590116 17352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:44:09.655361 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:10.021027 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:10.021333 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:10.021430 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:10.029074 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:10.029214 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:10.029330 17365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:44:10.040073 17365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:10.041419 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:10.042932 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:10.057096 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:10.057214 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:10.057188 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:10.058192 17365 server_base.cc:1034] running on GCE node
I20260619 18:44:10.058923 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:10.059999 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:10.061252 17365 hybrid_clock.cc:648] HybridClock initialized: now 1781894650061205 us; error 42 us; skew 500 ppm
I20260619 18:44:10.061612 17365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:10 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:10.065044 17365 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:44:10.066735 17365 webserver.cc:460] Webserver started at http://127.3.85.129:41779/ using document root <none> and password file <none>
I20260619 18:44:10.067485 17365 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:10.067617 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:10.067879 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:10.071077 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "02755be784b14113ac0e1c8842ed7314"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "9f0e0573e00500a706931441c4890464"
server_key_iv: "e73373302aff60eb411886a45b159359"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.071866 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "02755be784b14113ac0e1c8842ed7314"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "9f0e0573e00500a706931441c4890464"
server_key_iv: "e73373302aff60eb411886a45b159359"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.077622 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:44:10.081440 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.083133 17365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:10.083317 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "02755be784b14113ac0e1c8842ed7314"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "9f0e0573e00500a706931441c4890464"
server_key_iv: "e73373302aff60eb411886a45b159359"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.083524 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:10.102643 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:10.107550 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:10.107822 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:10.108816 17365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:10.110317 17365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:10.110399 17365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.110605 17365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:10.110659 17365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.147655 17365 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:39157
I20260619 18:44:10.147694 17492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:39157 every 8 connection(s)
I20260619 18:44:10.149111 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:10.154385 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17365
I20260619 18:44:10.154661 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:44:10.155311 3414 external_mini_cluster.cc:1428] Setting key b5242f59ca2f2a8d2cb93e6beea32e4e
Jun 19 18:44:10 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:10.174360 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.151972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:41701 (local address 127.3.85.190:46203)
0619 18:44:10.152476 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:10.152480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:10.153643 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 18:44:10.155956 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:10.155964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:10.156110 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:10.156239 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:10.158393 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.159689 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.161506 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.162309 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.162553 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.166699 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:10.166704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:10.166708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:10.166748 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:10.169775 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:10.171123 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:10.171129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:10.171132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:10.171299 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:10.172039 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:10.172044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:10.172046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:10.172346 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:10.172483 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:10.173585 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0619 18:44:10.174067 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260619 18:44:10.176060 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.152579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:41701)
0619 18:44:10.153429 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.153478 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:10.155579 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:10.156591 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:10.156603 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:10.157483 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:10.158148 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.158168 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.159944 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.159949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.161299 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.161314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.162706 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.162712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.162949 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.163764 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:10.163795 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:10.166238 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:10.170006 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:10.170031 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:10.170057 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:10.170929 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:10.171457 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:10.171465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:10.171472 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:10.171782 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:10.172510 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:10.172522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:10.173077 (+ 555us) client_negotiation.cc:769] Sending connection context
0619 18:44:10.173413 (+ 336us) client_negotiation.cc:240] Negotiation successful
0619 18:44:10.174365 (+ 952us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":347,"threads_started":1}
I20260619 18:44:10.178045 17493 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:10.178421 17493 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:10.179400 17493 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:44:10.181936 17304 ts_manager.cc:194] Registered new tserver with Master: 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157)
I20260619 18:44:10.183492 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:41701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:44:10.214056 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:10.563771 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:10.564157 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:10.564342 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:10.572629 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:10.572824 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:10.573026 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:44:10.584185 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:10.585647 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:10.587208 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:10.601114 17505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:10.601140 17507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:10.601538 17504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:10.601956 17499 server_base.cc:1034] running on GCE node
I20260619 18:44:10.602676 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:10.603703 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:10.604948 17499 hybrid_clock.cc:648] HybridClock initialized: now 1781894650604885 us; error 60 us; skew 500 ppm
I20260619 18:44:10.605360 17499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:10 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:10.608913 17499 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:44:10.610760 17499 webserver.cc:460] Webserver started at http://127.3.85.130:35149/ using document root <none> and password file <none>
I20260619 18:44:10.611562 17499 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:10.611704 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:10.612047 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:10.615341 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "c29b83535f40417685086498c3263609"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "c10de420e82cf32da6f03194e8067d5b"
server_key_iv: "aa6fefe76d62596bc95ebf9ec670a735"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.616242 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "c29b83535f40417685086498c3263609"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "c10de420e82cf32da6f03194e8067d5b"
server_key_iv: "aa6fefe76d62596bc95ebf9ec670a735"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.622324 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:44:10.626600 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.628302 17499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:10.628511 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "c29b83535f40417685086498c3263609"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "c10de420e82cf32da6f03194e8067d5b"
server_key_iv: "aa6fefe76d62596bc95ebf9ec670a735"
server_key_version: "encryptionkey@0"
I20260619 18:44:10.628731 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:10.649223 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:10.654095 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:10.654424 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:10.655561 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:10.657037 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:10.657114 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.657289 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:10.657341 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:10.692087 17499 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40109
I20260619 18:44:10.692134 17626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40109 every 8 connection(s)
I20260619 18:44:10.693560 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:10.698930 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17499
I20260619 18:44:10.699167 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:44:10.699726 3414 external_mini_cluster.cc:1428] Setting key eb27ce0ac206d9078cda1bbec22c5771
Jun 19 18:44:10 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:10.720826 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.696468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:49661 (local address 127.3.85.190:46203)
0619 18:44:10.697025 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:10.697030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:10.698006 (+ 976us) server_negotiation.cc:407] Connection header received
0619 18:44:10.700850 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:10.700856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:10.700975 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:10.701132 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:10.703416 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.704207 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.707180 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.708075 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.708278 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.713693 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:10.713704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:10.713709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:10.713766 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:10.716873 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:10.718134 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:10.718140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:10.718142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:10.718294 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:10.718895 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:10.718898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:10.718900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:10.719182 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:10.719316 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:10.720337 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0619 18:44:10.720545 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":314,"threads_started":1}
I20260619 18:44:10.722543 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:10.696977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:49661)
0619 18:44:10.697808 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:10.697853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:10.700441 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:10.701396 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:10.701406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:10.702212 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:10.703187 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.703219 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.704413 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.704422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.706874 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:10.706901 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:10.708430 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:10.708440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:10.708840 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:10.709800 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:10.709845 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:10.713281 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:10.717078 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:10.717093 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:10.717117 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:10.717972 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:10.718463 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:10.718471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:10.718476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:10.718769 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:10.719328 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:10.719340 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:10.719894 (+ 554us) client_negotiation.cc:769] Sending connection context
0619 18:44:10.720242 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 18:44:10.720891 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":318,"threads_started":1}
I20260619 18:44:10.724243 17627 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:10.724637 17627 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:10.725678 17627 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:44:10.727818 17304 ts_manager.cc:194] Registered new tserver with Master: c29b83535f40417685086498c3263609 (127.3.85.130:40109)
I20260619 18:44:10.728946 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:49661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:44:10.761765 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:11.112888 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:11.113261 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:11.113441 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:11.121171 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:11.121363 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:11.121572 17633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:44:11.131976 17633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:11.133610 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:11.135246 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:11.149663 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:11.149775 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:11.150784 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:11.151353 17633 server_base.cc:1034] running on GCE node
I20260619 18:44:11.152056 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:11.153167 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:11.154476 17633 hybrid_clock.cc:648] HybridClock initialized: now 1781894651154363 us; error 68 us; skew 500 ppm
I20260619 18:44:11.154839 17633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:11 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:11.158244 17633 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:44:11.159960 17633 webserver.cc:460] Webserver started at http://127.3.85.131:33133/ using document root <none> and password file <none>
I20260619 18:44:11.160771 17633 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:11.160931 17633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:11.161253 17633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:11.164544 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
format_stamp: "Formatted at 2026-06-19 18:44:11 on dist-test-slave-09fp"
server_key: "5ac2e61de2d13700e6535d0011454d13"
server_key_iv: "29c0c5485644fa16ba6a9b121e0b7d5c"
server_key_version: "encryptionkey@0"
I20260619 18:44:11.165459 17633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
format_stamp: "Formatted at 2026-06-19 18:44:11 on dist-test-slave-09fp"
server_key: "5ac2e61de2d13700e6535d0011454d13"
server_key_iv: "29c0c5485644fa16ba6a9b121e0b7d5c"
server_key_version: "encryptionkey@0"
I20260619 18:44:11.171481 17633 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:44:11.175614 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:11.177110 17633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:44:11.177321 17633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
format_stamp: "Formatted at 2026-06-19 18:44:11 on dist-test-slave-09fp"
server_key: "5ac2e61de2d13700e6535d0011454d13"
server_key_iv: "29c0c5485644fa16ba6a9b121e0b7d5c"
server_key_version: "encryptionkey@0"
I20260619 18:44:11.177568 17633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:11.187068 17493 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:11.211795 17633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:11.217044 17633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:11.217325 17633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:11.218430 17633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:11.220036 17633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:11.220121 17633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:11.220310 17633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:11.220371 17633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:11.254351 17633 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:33745
I20260619 18:44:11.254385 17760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:33745 every 8 connection(s)
I20260619 18:44:11.255837 17633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:11.256538 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17633
I20260619 18:44:11.256738 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:44:11.257306 3414 external_mini_cluster.cc:1428] Setting key 70e8cc37c8fb1d2acc79772a3b6f6739
Jun 19 18:44:11 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:11.278708 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.258660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:60523 (local address 127.3.85.190:46203)
0619 18:44:11.259687 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.259696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.260638 (+ 942us) server_negotiation.cc:407] Connection header received
0619 18:44:11.262519 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.262524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.262623 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.262719 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:11.264380 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.265209 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.267082 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.267473 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.267580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.271435 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:11.271444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:11.271448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:11.271494 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:11.275224 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.276117 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.276122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.276125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.276267 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.276753 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.276763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.276765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.277058 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:11.277233 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.278022 (+ 789us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.278281 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":754,"threads_started":1}
I20260619 18:44:11.279708 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.259396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:60523)
0619 18:44:11.260339 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.260405 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.262182 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.262957 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.262966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.263561 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:11.264181 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.264200 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.265382 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.265385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.266903 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.266917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.267698 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.267702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.267922 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.268721 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:11.268746 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:11.271143 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:11.275414 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:11.275423 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:11.275439 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:11.275959 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:11.276392 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:11.276403 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:11.276406 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:11.276621 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:11.277199 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:11.277208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:11.277618 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.277912 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.278539 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":365,"threads_started":1}
I20260619 18:44:11.281325 17761 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:11.281684 17761 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:11.282603 17761 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
I20260619 18:44:11.284261 17304 ts_manager.cc:194] Registered new tserver with Master: 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745)
I20260619 18:44:11.285288 17304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:60523
I20260619 18:44:11.286517 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:11.308350 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.291410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.3.85.190:46203)
0619 18:44:11.291601 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.291606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.292056 (+ 450us) server_negotiation.cc:407] Connection header received
0619 18:44:11.292203 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.292207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.292276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.292370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:11.293773 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.294602 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.296163 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.296569 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.296667 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.302683 (+ 6016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:11.302693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:11.302697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:11.302741 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:11.305437 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.306252 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.306257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.306260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.306396 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.306900 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.306904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.306906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.307255 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:11.307444 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.307896 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.308106 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:44:11.313131 17304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:11.320422 17304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:11.349119 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.334671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.3.85.131:33745)
0619 18:44:11.335736 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.335748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.336648 (+ 900us) server_negotiation.cc:407] Connection header received
0619 18:44:11.337151 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.337163 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.337530 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.337740 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:11.339145 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.341376 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.345420 (+ 4044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.347292 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.347419 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.347710 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.348467 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.348679 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":693,"threads_started":1}
I20260619 18:44:11.349152 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.334174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:33745 (local address 127.0.0.1:35188)
0619 18:44:11.336449 (+ 2275us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.336515 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.336849 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.338225 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.338236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.338285 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:11.338851 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.338868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.341682 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.341691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.345115 (+ 3424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.345148 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.347917 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.347925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.348144 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.348199 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.348379 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.348654 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1942,"thread_start_us":12614,"threads_started":1}
I20260619 18:44:11.353880 17696 tablet_service.cc:1467] Processing CreateTablet for tablet 871a5a4338e648e7a5b53625587e142b (DEFAULT_TABLE table=test-table [id=db831179b64f4d418c6eb94422e503d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:11.355763 17696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a5a4338e648e7a5b53625587e142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:11.356139 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.336747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35604)
0619 18:44:11.347667 (+ 10920us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.347688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.347816 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.348782 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.348789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.348829 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:11.349315 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.349329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.351107 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.351112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.352941 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.352957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.355152 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.355159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.355365 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.355402 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.355502 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.355667 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10791,"mutex_wait_us":80,"thread_start_us":10031,"threads_started":1}
I20260619 18:44:11.356689 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.334609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.3.85.130:40109)
0619 18:44:11.335390 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.335403 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.347291 (+ 11888us) server_negotiation.cc:407] Connection header received
0619 18:44:11.347595 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.347606 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.347893 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.348104 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:11.349090 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.350703 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.353713 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.355257 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.355394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.355588 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.356163 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.356360 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":395,"threads_started":1}
I20260619 18:44:11.357132 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.337101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.3.85.129:39157)
0619 18:44:11.346567 (+ 9466us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.346584 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.347962 (+ 1378us) server_negotiation.cc:407] Connection header received
0619 18:44:11.348082 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.348090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.348332 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.348514 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:11.349638 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.350928 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.353141 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.354587 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.354735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.354978 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.356498 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.356707 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9228,"thread_start_us":9055,"threads_started":1}
I20260619 18:44:11.356581 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.334300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40109 (local address 127.0.0.1:35884)
0619 18:44:11.347125 (+ 12825us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.347143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.347340 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.348420 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.348424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.348454 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:11.348848 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.348861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.351132 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.351137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.353509 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.353526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.355735 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.355741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.355915 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.355947 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.356059 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.356231 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12690,"mutex_wait_us":1,"thread_start_us":1641,"threads_started":1}
I20260619 18:44:11.361390 17562 tablet_service.cc:1467] Processing CreateTablet for tablet 871a5a4338e648e7a5b53625587e142b (DEFAULT_TABLE table=test-table [id=db831179b64f4d418c6eb94422e503d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:11.361390 17428 tablet_service.cc:1467] Processing CreateTablet for tablet 871a5a4338e648e7a5b53625587e142b (DEFAULT_TABLE table=test-table [id=db831179b64f4d418c6eb94422e503d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:11.363023 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a5a4338e648e7a5b53625587e142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:11.363056 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871a5a4338e648e7a5b53625587e142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:11.368981 17780 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Bootstrap starting.
I20260619 18:44:11.373422 17780 tablet_bootstrap.cc:654] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:11.374738 17780 log.cc:824] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:11.377900 17780 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: No bootstrap required, opened a new log
I20260619 18:44:11.378360 17780 ts_tablet_manager.cc:1397] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 18:44:11.380107 17782 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Bootstrap starting.
I20260619 18:44:11.382402 17783 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Bootstrap starting.
I20260619 18:44:11.386526 17783 tablet_bootstrap.cc:654] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:11.388162 17783 log.cc:824] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:11.389725 17782 tablet_bootstrap.cc:654] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:11.390117 17780 raft_consensus.cc:348] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.390663 17780 raft_consensus.cc:374] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:11.390792 17780 raft_consensus.cc:729] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a89ebb2c714ea8a2bea890bd5e897a, State: Initialized, Role: FOLLOWER
I20260619 18:44:11.391145 17782 log.cc:824] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:11.391489 17780 consensus_queue.cc:260] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.392143 17783 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: No bootstrap required, opened a new log
I20260619 18:44:11.392539 17783 ts_tablet_manager.cc:1397] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260619 18:44:11.393580 17780 ts_tablet_manager.cc:1428] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 18:44:11.394170 17782 tablet_bootstrap.cc:492] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: No bootstrap required, opened a new log
I20260619 18:44:11.394441 17782 ts_tablet_manager.cc:1397] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260619 18:44:11.394492 17761 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:11.402273 17783 raft_consensus.cc:348] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.402789 17783 raft_consensus.cc:374] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:11.402904 17783 raft_consensus.cc:729] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02755be784b14113ac0e1c8842ed7314, State: Initialized, Role: FOLLOWER
I20260619 18:44:11.403527 17783 consensus_queue.cc:260] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.403939 17782 raft_consensus.cc:348] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.404347 17782 raft_consensus.cc:374] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:11.404472 17782 raft_consensus.cc:729] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29b83535f40417685086498c3263609, State: Initialized, Role: FOLLOWER
I20260619 18:44:11.405366 17783 ts_tablet_manager.cc:1428] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 18:44:11.405187 17782 consensus_queue.cc:260] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.406288 17627 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:11.406967 17782 ts_tablet_manager.cc:1428] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260619 18:44:11.445845 17628 tablet.cc:2369] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:11.490813 17788 raft_consensus.cc:482] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:11.491164 17788 raft_consensus.cc:504] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.493656 17788 leader_election.cc:288] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745), 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157)
I20260619 18:44:11.502347 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.494528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:55805 (local address 127.3.85.131:33745)
0619 18:44:11.494733 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.494740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.494770 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:44:11.494970 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.494978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.495089 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.495190 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:11.495927 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.496891 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.499603 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.501026 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.501240 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.501335 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.501939 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.502083 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:44:11.502362 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.494437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:45989 (local address 127.3.85.129:39157)
0619 18:44:11.494745 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.494760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.495520 (+ 760us) server_negotiation.cc:407] Connection header received
0619 18:44:11.495697 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.495705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.495803 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.495925 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:11.496574 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.497532 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.500033 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.500849 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.501075 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.501160 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.501558 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.501729 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:44:11.502427 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.493915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:33745 (local address 127.3.85.130:55805)
0619 18:44:11.494627 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.494650 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.494818 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.495371 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.495375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.495406 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:11.495785 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.495795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.497054 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.497061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.499400 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.499420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.501511 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.501518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.501704 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.501738 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.501873 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.502033 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":22,"thread_start_us":898,"threads_started":1}
I20260619 18:44:11.503268 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.493915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.3.85.130:45989)
0619 18:44:11.495381 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:11.495398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:11.495548 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:11.495951 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:11.495956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:11.495986 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:11.496375 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.496391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.497678 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.497684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.499858 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:11.499871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.501105 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.501112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:11.501299 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.501340 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:44:11.502681 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0619 18:44:11.502877 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":69,"thread_start_us":1103,"threads_started":1}
I20260619 18:44:11.503489 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a5a4338e648e7a5b53625587e142b" candidate_uuid: "c29b83535f40417685086498c3263609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" is_pre_election: true
I20260619 18:44:11.504070 17716 raft_consensus.cc:2454] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29b83535f40417685086498c3263609 in term 0.
I20260619 18:44:11.504366 17448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a5a4338e648e7a5b53625587e142b" candidate_uuid: "c29b83535f40417685086498c3263609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02755be784b14113ac0e1c8842ed7314" is_pre_election: true
I20260619 18:44:11.504774 17517 leader_election.cc:302] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a89ebb2c714ea8a2bea890bd5e897a, c29b83535f40417685086498c3263609; no voters:
I20260619 18:44:11.504803 17448 raft_consensus.cc:2454] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c29b83535f40417685086498c3263609 in term 0.
I20260619 18:44:11.505189 17788 raft_consensus.cc:2781] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:11.505347 17788 raft_consensus.cc:482] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:11.505451 17788 raft_consensus.cc:3037] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Advancing to term 1
W20260619 18:44:11.507759 17762 tablet.cc:2369] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:11.509208 17788 raft_consensus.cc:504] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.510125 17788 leader_election.cc:288] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [CANDIDATE]: Term 1 election: Requested vote from peers 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745), 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157)
I20260619 18:44:11.510700 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a5a4338e648e7a5b53625587e142b" candidate_uuid: "c29b83535f40417685086498c3263609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
I20260619 18:44:11.510826 17448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871a5a4338e648e7a5b53625587e142b" candidate_uuid: "c29b83535f40417685086498c3263609" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02755be784b14113ac0e1c8842ed7314"
I20260619 18:44:11.510965 17716 raft_consensus.cc:3037] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:11.511071 17448 raft_consensus.cc:3037] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:11.514621 17448 raft_consensus.cc:2454] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29b83535f40417685086498c3263609 in term 1.
I20260619 18:44:11.514788 17716 raft_consensus.cc:2454] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c29b83535f40417685086498c3263609 in term 1.
I20260619 18:44:11.515175 17515 leader_election.cc:302] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02755be784b14113ac0e1c8842ed7314, c29b83535f40417685086498c3263609; no voters:
I20260619 18:44:11.515566 17788 raft_consensus.cc:2781] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:11.516125 17788 raft_consensus.cc:686] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 1 LEADER]: Becoming Leader. State: Replica: c29b83535f40417685086498c3263609, State: Running, Role: LEADER
I20260619 18:44:11.516734 17788 consensus_queue.cc:237] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:11.522934 17303 catalog_manager.cc:5495] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 reported cstate change: term changed from 0 to 1, leader changed from <none> to c29b83535f40417685086498c3263609 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "c29b83535f40417685086498c3263609" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } health_report { overall_health: UNKNOWN } } }
I20260619 18:44:11.553927 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.548677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.3.85.130:40109)
0619 18:44:11.548870 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.548875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.548892 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:44:11.548992 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.548995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.549091 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.549223 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:11.549568 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.550319 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.551777 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.552195 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.552293 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.552848 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:11.553405 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:11.553517 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.553627 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.553703 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:44:11.562994 17448 raft_consensus.cc:1261] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Refusing update from remote peer c29b83535f40417685086498c3263609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:11.562997 17716 raft_consensus.cc:1261] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Refusing update from remote peer c29b83535f40417685086498c3263609: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:11.564103 17791 consensus_queue.cc:1035] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:11.564529 17788 consensus_queue.cc:1035] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:11.579818 17795 mvcc.cc:205] Tried to move back new op lower bound from 7298640492790104064 to 7298640492617940992. Current Snapshot: MvccSnapshot[applied={T|T < 7298640492790104064}]
I20260619 18:44:11.593689 17304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:11.593982 17304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:11.598585 17304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Sending DeleteTablet for 3 replicas of tablet 871a5a4338e648e7a5b53625587e142b
I20260619 18:44:11.600144 17428 tablet_service.cc:1514] Processing DeleteTablet for tablet 871a5a4338e648e7a5b53625587e142b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35604
I20260619 18:44:11.600155 17696 tablet_service.cc:1514] Processing DeleteTablet for tablet 871a5a4338e648e7a5b53625587e142b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35188
I20260619 18:44:11.600569 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17365
I20260619 18:44:11.601271 17562 tablet_service.cc:1514] Processing DeleteTablet for tablet 871a5a4338e648e7a5b53625587e142b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:11 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35884
I20260619 18:44:11.615170 17804 tablet_replica.cc:330] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: stopping tablet replica
I20260619 18:44:11.616777 17804 raft_consensus.cc:2229] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:11.617835 17807 tablet_replica.cc:330] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: stopping tablet replica
I20260619 18:44:11.618252 17807 raft_consensus.cc:2229] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:11.619050 17807 raft_consensus.cc:2258] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:11.619669 17804 raft_consensus.cc:2258] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:11.620229 17805 tablet_replica.cc:330] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: stopping tablet replica
I20260619 18:44:11.620666 17805 raft_consensus.cc:2229] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:11.621174 17805 raft_consensus.cc:2258] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:11.621791 17807 ts_tablet_manager.cc:1905] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:11.623560 17804 ts_tablet_manager.cc:1905] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:11.623566 17805 ts_tablet_manager.cc:1905] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:11.629809 17807 ts_tablet_manager.cc:1918] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:11.630060 17807 log.cc:1196] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/871a5a4338e648e7a5b53625587e142b
I20260619 18:44:11.630790 17807 ts_tablet_manager.cc:1939] T 871a5a4338e648e7a5b53625587e142b P c29b83535f40417685086498c3263609: Deleting consensus metadata
I20260619 18:44:11.630918 17805 ts_tablet_manager.cc:1918] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:11.631107 17805 log.cc:1196] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/871a5a4338e648e7a5b53625587e142b
I20260619 18:44:11.631773 17805 ts_tablet_manager.cc:1939] T 871a5a4338e648e7a5b53625587e142b P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting consensus metadata
I20260619 18:44:11.633013 17288 catalog_manager.cc:4841] TS c29b83535f40417685086498c3263609 (127.3.85.130:40109): tablet 871a5a4338e648e7a5b53625587e142b (table test-table [id=db831179b64f4d418c6eb94422e503d0]) successfully deleted
Jun 19 18:44:11 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:44:11.634459 17290 catalog_manager.cc:4841] TS 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745): tablet 871a5a4338e648e7a5b53625587e142b (table test-table [id=db831179b64f4d418c6eb94422e503d0]) successfully deleted
I20260619 18:44:11.636615 17804 ts_tablet_manager.cc:1918] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:11.636813 17804 log.cc:1196] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/871a5a4338e648e7a5b53625587e142b
I20260619 18:44:11.637516 17804 ts_tablet_manager.cc:1939] T 871a5a4338e648e7a5b53625587e142b P 02755be784b14113ac0e1c8842ed7314: Deleting consensus metadata
I20260619 18:44:11.639740 17288 catalog_manager.cc:4841] TS 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157): tablet 871a5a4338e648e7a5b53625587e142b (table test-table [id=db831179b64f4d418c6eb94422e503d0]) successfully deleted
I20260619 18:44:11.642705 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.601815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.3.85.129:39157)
0619 18:44:11.602087 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.602098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.611774 (+ 9676us) server_negotiation.cc:407] Connection header received
0619 18:44:11.612433 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.612443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.612582 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.612676 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:11.617198 (+ 4522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.618365 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.623906 (+ 5541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.624584 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.624708 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.635834 (+ 11126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:11.635852 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:11.635867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:11.635914 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:11.639251 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.640258 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.640266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.640269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.640419 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.640919 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.640926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.640930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.641319 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:11.641546 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.642068 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.642300 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:44:11.643544 17488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:11.734728 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17365
I20260619 18:44:11.753485 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17499
Jun 19 18:44:11 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:44:11.771145 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.754327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.3.85.130:40109)
0619 18:44:11.754549 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.754556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.754577 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:44:11.754787 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.754795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.754867 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.755006 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:11.756365 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.757217 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.758751 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.759215 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.759376 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.764084 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:11.764102 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:11.764118 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:11.764163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:11.767884 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.768800 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.768809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.768811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.768986 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.769411 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.769415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.769417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.769811 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:11.770186 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.770514 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.770781 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:44:11.771889 17622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:11.884908 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17499
I20260619 18:44:11.904278 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17633
Jun 19 18:44:11 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:44:11.920090 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:11.905090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.3.85.131:33745)
0619 18:44:11.905297 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:11.905305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:11.905327 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:44:11.905560 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:11.905565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:11.905646 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:11.905768 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:11.907212 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.907937 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.909178 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:11.909584 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:11.909734 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:11.914337 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:11.914354 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:11.914370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:11.914409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:11.917315 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.918151 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.918158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.918161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.918308 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:11.918771 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:11.918775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:11.918777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:11.919145 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:11.919380 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:11.919649 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 18:44:11.919823 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:44:11.920776 17756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:12.013041 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17633
I20260619 18:44:12.030185 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17272
I20260619 18:44:12.031495 17334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:12.143966 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17272
I20260619 18:44:12.166023 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--webserver_interface=127.3.85.190
--webserver_port=42733
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:12.523154 17812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:12.523478 17812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:12.523666 17812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:12.531826 17812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:12.531973 17812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:12.532080 17812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:12.532171 17812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:12.532253 17812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:12.543572 17812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=42733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:12.545068 17812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:12.546530 17812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:12.571470 17818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:12.571470 17821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:12.571509 17819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:12.572734 17812 server_base.cc:1034] running on GCE node
I20260619 18:44:12.573717 17812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:12.575559 17812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:12.576913 17812 hybrid_clock.cc:648] HybridClock initialized: now 1781894652576834 us; error 63 us; skew 500 ppm
I20260619 18:44:12.577468 17812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:12 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:12.581493 17812 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:12.583420 17812 webserver.cc:460] Webserver started at http://127.3.85.190:42733/ using document root <none> and password file <none>
I20260619 18:44:12.584259 17812 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:12.584437 17812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:12.591550 17812 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:44:12.594964 17828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:12.596565 17812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:44:12.596793 17812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "afc045bcdf93480187bb2054ac9652c1"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-09fp"
server_key: "df9cd311e869518cb50beeac966f36db"
server_key_iv: "d13e5887a836c3ad0418151ff747f709"
server_key_version: "encryptionkey@0"
I20260619 18:44:12.598011 17812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:12.619509 17812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:12.624414 17812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:12.624733 17812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:12.645462 17812 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46203
I20260619 18:44:12.645483 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46203 every 8 connection(s)
I20260619 18:44:12.647264 17812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:12.652220 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17812
I20260619 18:44:12.653056 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:39157
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=41779
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:12.655105 17880 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:44:12.658872 17880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap starting.
I20260619 18:44:12.680272 17880 log.cc:824] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:12.694016 17880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:44:12.694713 17880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap complete.
I20260619 18:44:12.705179 17880 raft_consensus.cc:348] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:12.706291 17880 raft_consensus.cc:729] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Initialized, Role: FOLLOWER
I20260619 18:44:12.707248 17880 consensus_queue.cc:260] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:12.707599 17880 raft_consensus.cc:388] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:12.707751 17880 raft_consensus.cc:482] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:12.707975 17880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:44:12.714581 17880 raft_consensus.cc:504] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:12.715348 17880 leader_election.cc:302] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afc045bcdf93480187bb2054ac9652c1; no voters:
I20260619 18:44:12.716641 17880 leader_election.cc:288] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:44:12.716794 17884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:44:12.719465 17880 sys_catalog.cc:564] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:12.724009 17884 raft_consensus.cc:686] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 LEADER]: Becoming Leader. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Running, Role: LEADER
I20260619 18:44:12.724834 17884 consensus_queue.cc:237] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:12.727897 17885 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:12.728410 17885 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:12.728994 17886 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afc045bcdf93480187bb2054ac9652c1. Latest consensus state: current_term: 2 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:12.729344 17886 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:12.732765 17893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:12.743958 17893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:12.746035 17893 catalog_manager.cc:1259] Loaded cluster ID: 2589391bb5984c47957ac33edf1bdbad
I20260619 18:44:12.746207 17893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:12.750824 17893 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 18:44:12.781141 17893 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 18:44:12.791455 17893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:12.792876 17893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Loaded TSK: 0
I20260619 18:44:12.794202 17893 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:44:13.056140 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:13.056468 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:13.056634 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:13.064530 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:13.064671 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:13.064850 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:44:13.075350 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:39157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=41779
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:13.076771 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:13.078274 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:13.094614 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:13.094660 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:13.095350 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:13.095960 17882 server_base.cc:1034] running on GCE node
I20260619 18:44:13.096879 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:13.098321 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:13.099627 17882 hybrid_clock.cc:648] HybridClock initialized: now 1781894653099564 us; error 76 us; skew 500 ppm
I20260619 18:44:13.100085 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:13 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:13.110726 17882 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:44:13.112943 17882 webserver.cc:460] Webserver started at http://127.3.85.129:41779/ using document root <none> and password file <none>
I20260619 18:44:13.114058 17882 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:13.114241 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:13.122519 17882 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:44:13.126334 17917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.128340 17882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:13.128548 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "02755be784b14113ac0e1c8842ed7314"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "9f0e0573e00500a706931441c4890464"
server_key_iv: "e73373302aff60eb411886a45b159359"
server_key_version: "encryptionkey@0"
I20260619 18:44:13.129937 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:13.155253 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:13.160115 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:13.160442 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:13.161556 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:13.163054 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:13.163203 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.163348 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:13.163447 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.199954 17882 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:39157
I20260619 18:44:13.200008 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:39157 every 8 connection(s)
I20260619 18:44:13.201575 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:13.204437 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 17882
I20260619 18:44:13.205338 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:40109
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=35149
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:13 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:13.232609 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:13.205005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:59497 (local address 127.3.85.190:46203)
0619 18:44:13.205755 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:13.205780 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:13.206822 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 18:44:13.210738 (+ 3916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:13.210755 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:13.211200 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:13.211579 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:13.214074 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.216357 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.218132 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.219364 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.219588 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:13.223740 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:13.223755 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:13.223781 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:13.223842 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:13.227633 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:13.228531 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:13.228540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:13.228546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:13.228727 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:13.229199 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:13.229205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:13.229210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:13.229720 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:13.229937 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:13.230798 (+ 861us) server_negotiation.cc:299] Negotiation successful
0619 18:44:13.231215 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":305,"threads_started":1}
I20260619 18:44:13.232604 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:13.205314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:59497)
0619 18:44:13.206571 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:13.206635 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:13.209800 (+ 3165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:13.211856 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:13.211871 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:13.212793 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:13.213696 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:13.213721 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.216610 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.216615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:13.217920 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:13.217936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.219641 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.219647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:13.219921 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:13.220708 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:13.220734 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:13.223364 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:13.227843 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:13.227852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:13.227875 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:13.228356 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:13.228862 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:13.228867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:13.228871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:13.229073 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:13.229878 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:13.229887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:13.230235 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 18:44:13.230573 (+ 338us) client_negotiation.cc:240] Negotiation successful
0619 18:44:13.231194 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":565,"threads_started":1}
I20260619 18:44:13.235710 18030 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:13.236061 18030 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:13.236923 18030 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
I20260619 18:44:13.239370 17845 ts_manager.cc:194] Registered new tserver with Master: 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157)
I20260619 18:44:13.240897 17845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:59497
W20260619 18:44:13.564220 18034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:13.564589 18034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:13.564769 18034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:13.572482 18034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:13.572645 18034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:13.572851 18034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:44:13.584095 18034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:40109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=35149
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:13.585536 18034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:13.586934 18034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:13.601725 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:13.601895 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:13.601895 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:13.603344 18034 server_base.cc:1034] running on GCE node
I20260619 18:44:13.604143 18034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:13.605265 18034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:13.606590 18034 hybrid_clock.cc:648] HybridClock initialized: now 1781894653606530 us; error 48 us; skew 500 ppm
I20260619 18:44:13.606964 18034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:13 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:13.610612 18034 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:44:13.612277 18034 webserver.cc:460] Webserver started at http://127.3.85.130:35149/ using document root <none> and password file <none>
I20260619 18:44:13.613049 18034 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:13.613226 18034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:13.619532 18034 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:44:13.622692 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.624295 18034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:44:13.624544 18034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "c29b83535f40417685086498c3263609"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "c10de420e82cf32da6f03194e8067d5b"
server_key_iv: "aa6fefe76d62596bc95ebf9ec670a735"
server_key_version: "encryptionkey@0"
I20260619 18:44:13.625626 18034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:13.637691 18034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:13.642088 18034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:13.642406 18034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:13.643517 18034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:13.645139 18034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:13.645251 18034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.645356 18034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:13.645440 18034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:13.680346 18034 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40109
I20260619 18:44:13.680459 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40109 every 8 connection(s)
I20260619 18:44:13.681900 18034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:13.684845 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18034
I20260619 18:44:13.685490 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:33745
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=33133
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:13 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894653, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:13.706609 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:13.684516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:53263 (local address 127.3.85.190:46203)
0619 18:44:13.684760 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:13.684768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:13.686193 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 18:44:13.688217 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:13.688226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:13.688348 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:13.688422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:13.690885 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.691911 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.693907 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.694740 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.694907 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:13.698660 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:13.698671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:13.698679 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:13.698731 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:13.702557 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:13.703643 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:13.703647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:13.703650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:13.703804 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:13.704592 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:13.704603 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:13.704605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:13.704948 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:13.705126 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:13.705934 (+ 808us) server_negotiation.cc:299] Negotiation successful
0619 18:44:13.706214 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:44:13.707659 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:13.685086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:53263)
0619 18:44:13.685980 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:13.686031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:13.687853 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:13.688992 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:13.689006 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:13.689797 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:13.690622 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:13.690657 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.692134 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.692141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:13.693702 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:13.693717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:13.695021 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:13.695027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:13.695253 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:13.696031 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:13.696057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:13.698346 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:13.702784 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:13.702793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:13.702853 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:13.703420 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:13.704152 (+ 732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:13.704160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:13.704164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:13.704441 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:13.705109 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:13.705119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:13.705516 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 18:44:13.705807 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:44:13.706441 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260619 18:44:13.709606 18162 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:13.709970 18162 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:13.710793 18162 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
I20260619 18:44:13.712505 17845 ts_manager.cc:194] Registered new tserver with Master: c29b83535f40417685086498c3263609 (127.3.85.130:40109)
I20260619 18:44:13.713672 17845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:53263
W20260619 18:44:14.033414 18165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:14.033727 18165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:14.033825 18165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:14.041406 18165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:14.041533 18165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:14.041651 18165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:44:14.052168 18165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=33133
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:14.053552 18165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:14.054939 18165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:14.069193 18171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:14.069185 18170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:14.069180 18173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:14.070091 18165 server_base.cc:1034] running on GCE node
I20260619 18:44:14.070849 18165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:14.071939 18165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:14.073189 18165 hybrid_clock.cc:648] HybridClock initialized: now 1781894654073116 us; error 74 us; skew 500 ppm
I20260619 18:44:14.073536 18165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:14 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:14.077018 18165 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:44:14.078776 18165 webserver.cc:460] Webserver started at http://127.3.85.131:33133/ using document root <none> and password file <none>
I20260619 18:44:14.079555 18165 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:14.079720 18165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:14.086091 18165 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 18:44:14.089838 18180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:14.091529 18165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:14.091749 18165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
format_stamp: "Formatted at 2026-06-19 18:44:11 on dist-test-slave-09fp"
server_key: "5ac2e61de2d13700e6535d0011454d13"
server_key_iv: "29c0c5485644fa16ba6a9b121e0b7d5c"
server_key_version: "encryptionkey@0"
I20260619 18:44:14.092813 18165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:14.112784 18165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:14.117717 18165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:14.118034 18165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:14.119194 18165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:14.120791 18165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:14.120921 18165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:14.121093 18165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:14.121160 18165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:14.157291 18165 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:33745
I20260619 18:44:14.157346 18292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:33745 every 8 connection(s)
I20260619 18:44:14.159152 18165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:14.163868 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18165
Jun 19 18:44:14 dist-test-slave-09fp krb5kdc[17259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:14.183739 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.165180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.3.85.190:46203)
0619 18:44:14.165637 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.165641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.165660 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:44:14.165807 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.165811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.165919 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.166034 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:14.168127 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.169546 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.171602 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.172189 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.172338 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.175874 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:14.175883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:14.175889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:14.175937 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:14.180412 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.181286 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.181290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.181292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.181426 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.181890 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.181893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.181895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.182372 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:14.182634 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.183034 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.183338 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":7808,"thread_start_us":314,"threads_started":1}
I20260619 18:44:14.184840 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.161919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:46271 (local address 127.3.85.190:46203)
0619 18:44:14.162161 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.162168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.163471 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 18:44:14.165882 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.165886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.165932 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.166034 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:14.168326 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.169546 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.171934 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.172635 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.172743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.177284 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:14.177292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:14.177297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:14.177346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:14.180985 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.182069 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.182074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.182077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.182195 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.182757 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.182760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.182762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.183109 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:14.183289 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.184260 (+ 971us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.184496 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":96,"spinlock_wait_cycles":37760}
I20260619 18:44:14.186269 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.162435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:46271)
0619 18:44:14.163244 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.163291 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.165572 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.166371 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.166386 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.167241 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:14.168156 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.168178 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.169761 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.169766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.171798 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.171812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.172891 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.172896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.173263 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.173887 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:14.173919 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:14.176998 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:14.181200 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:14.181216 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:14.181238 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:14.181938 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:14.182313 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:14.182316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:14.182318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:14.182618 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:14.183261 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:14.183271 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:14.183766 (+ 495us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.184192 (+ 426us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.184758 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260619 18:44:14.187762 18293 heartbeater.cc:347] Connected to a master server at 127.3.85.190:46203
I20260619 18:44:14.188055 18293 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:14.188854 18293 heartbeater.cc:511] Master 127.3.85.190:46203 requested a full tablet report, sending...
I20260619 18:44:14.190434 17845 ts_manager.cc:194] Registered new tserver with Master: 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745)
I20260619 18:44:14.191308 17845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:46271
I20260619 18:44:14.200270 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:14.208375 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.201399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.3.85.190:46203)
0619 18:44:14.201576 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.201580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.202145 (+ 565us) server_negotiation.cc:407] Connection header received
0619 18:44:14.202421 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.202425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.202649 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.202829 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:14.203343 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.204197 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.205894 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.206521 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.206676 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.207232 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:14.207754 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:14.207929 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.207994 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.208086 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:44:14.209539 17845 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:14.217263 17845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:14.244699 18030 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:14.252302 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.231631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.3.85.130:40109)
0619 18:44:14.242797 (+ 11166us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.242811 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.242859 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:44:14.243008 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.243022 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.243437 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.243759 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:14.244983 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.246572 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.249316 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.251031 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.251187 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.251427 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.251785 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.251958 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10897,"thread_start_us":10754,"threads_started":1}
I20260619 18:44:14.252377 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.231083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40109 (local address 127.0.0.1:35916)
0619 18:44:14.232865 (+ 1782us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.232883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.232989 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.244163 (+ 11174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.244185 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.244226 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:14.244722 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.244738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.246918 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.246923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.249067 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.249080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.251322 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.251328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.251530 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.251576 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.251735 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.251951 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1656,"mutex_wait_us":4,"thread_start_us":1480,"threads_started":1}
I20260619 18:44:14.254036 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.231072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:33745 (local address 127.0.0.1:35206)
0619 18:44:14.232145 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.232195 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.232382 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.244967 (+ 12585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.244972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.245002 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:14.245373 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.245391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.248186 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.248192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.251021 (+ 2829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.251036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.253051 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.253061 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.253273 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.253319 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.253419 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.253562 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":292,"thread_start_us":665,"threads_started":1}
I20260619 18:44:14.254078 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.233110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35628)
0619 18:44:14.243301 (+ 10191us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.243332 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.243539 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.246867 (+ 3328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.246873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.246907 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:14.247404 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.247417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.249381 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.249386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.251164 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.251181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.253086 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.253093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.253321 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.253346 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.253449 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.253605 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9993,"mutex_wait_us":33,"thread_start_us":9827,"threads_started":1}
I20260619 18:44:14.255118 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.231762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.3.85.131:33745)
0619 18:44:14.244264 (+ 12502us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.244275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.244318 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:44:14.244422 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.244430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.244691 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.245647 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:14.246094 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.247993 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.251226 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.252727 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.252942 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.253212 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.253490 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.253658 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12293,"thread_start_us":12119,"threads_started":1}
I20260619 18:44:14.256253 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.233702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.3.85.129:39157)
0619 18:44:14.245885 (+ 12183us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.245896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.245925 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:44:14.246142 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.246151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.246586 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.247618 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:14.247876 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.249210 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.251383 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.252814 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.254210 (+ 1396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.254850 (+ 640us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.254929 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.255120 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11986,"thread_start_us":11698,"threads_started":1}
I20260619 18:44:14.257184 18097 tablet_service.cc:1467] Processing CreateTablet for tablet de1a9956877b4953933a12862cfe5b84 (DEFAULT_TABLE table=test-table [id=d2e17bdf07ec413898e623bbde53bb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:14.259052 18097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1a9956877b4953933a12862cfe5b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:14.259418 17965 tablet_service.cc:1467] Processing CreateTablet for tablet de1a9956877b4953933a12862cfe5b84 (DEFAULT_TABLE table=test-table [id=d2e17bdf07ec413898e623bbde53bb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:14.259431 18228 tablet_service.cc:1467] Processing CreateTablet for tablet de1a9956877b4953933a12862cfe5b84 (DEFAULT_TABLE table=test-table [id=d2e17bdf07ec413898e623bbde53bb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:14.261004 18228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1a9956877b4953933a12862cfe5b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:14.261039 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1a9956877b4953933a12862cfe5b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:14.274055 18309 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Bootstrap starting.
I20260619 18:44:14.278049 18309 tablet_bootstrap.cc:654] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:14.279538 18309 log.cc:824] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:14.282189 18310 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Bootstrap starting.
I20260619 18:44:14.283290 18309 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: No bootstrap required, opened a new log
I20260619 18:44:14.283501 18311 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Bootstrap starting.
I20260619 18:44:14.283775 18309 ts_tablet_manager.cc:1397] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:44:14.286247 18310 tablet_bootstrap.cc:654] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:14.287354 18311 tablet_bootstrap.cc:654] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:14.287613 18310 log.cc:824] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:14.288769 18311 log.cc:824] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:14.293558 18311 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: No bootstrap required, opened a new log
I20260619 18:44:14.293673 18310 tablet_bootstrap.cc:492] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: No bootstrap required, opened a new log
I20260619 18:44:14.293922 18311 ts_tablet_manager.cc:1397] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260619 18:44:14.293967 18310 ts_tablet_manager.cc:1397] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260619 18:44:14.296249 18309 raft_consensus.cc:348] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.296828 18309 raft_consensus.cc:374] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:14.296977 18309 raft_consensus.cc:729] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02755be784b14113ac0e1c8842ed7314, State: Initialized, Role: FOLLOWER
I20260619 18:44:14.297725 18309 consensus_queue.cc:260] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.299721 18309 ts_tablet_manager.cc:1428] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 18:44:14.303979 18311 raft_consensus.cc:348] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.304430 18311 raft_consensus.cc:374] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:14.304564 18311 raft_consensus.cc:729] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a89ebb2c714ea8a2bea890bd5e897a, State: Initialized, Role: FOLLOWER
I20260619 18:44:14.304503 18310 raft_consensus.cc:348] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.305097 18310 raft_consensus.cc:374] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:14.305212 18310 raft_consensus.cc:729] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29b83535f40417685086498c3263609, State: Initialized, Role: FOLLOWER
I20260619 18:44:14.305272 18311 consensus_queue.cc:260] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.305820 18310 consensus_queue.cc:260] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.306847 18293 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:14.307374 18311 ts_tablet_manager.cc:1428] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260619 18:44:14.311343 18162 heartbeater.cc:503] Master 127.3.85.190:46203 was elected leader, sending a full tablet report...
I20260619 18:44:14.311968 18310 ts_tablet_manager.cc:1428] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20260619 18:44:14.343281 18315 raft_consensus.cc:482] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:14.343592 18315 raft_consensus.cc:504] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.345430 18315 leader_election.cc:288] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c29b83535f40417685086498c3263609 (127.3.85.130:40109), 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745)
I20260619 18:44:14.353463 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.346055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:56577 (local address 127.3.85.131:33745)
0619 18:44:14.346332 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.346340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.347380 (+ 1040us) server_negotiation.cc:407] Connection header received
0619 18:44:14.347644 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.347647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.347707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.347826 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:14.348534 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.349279 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.351570 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.352382 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.352573 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.352683 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.353030 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.353172 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:44:14.353494 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.345964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:34359 (local address 127.3.85.130:40109)
0619 18:44:14.346249 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.346259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.347356 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 18:44:14.347585 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.347593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.347682 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.347755 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:14.348474 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.349246 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.351545 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.352388 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.352597 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.352682 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.353074 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.353225 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:44:14.353646 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.345749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:33745 (local address 127.3.85.129:56577)
0619 18:44:14.347220 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.347241 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.347468 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.347870 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.347874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.347905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:14.348338 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.348348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.349467 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.349471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.351371 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.351385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.352650 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.352657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.352854 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.352881 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.352965 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.353137 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":60,"spinlock_wait_cycles":45440,"thread_start_us":997,"threads_started":1}
I20260619 18:44:14.353905 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.345676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40109 (local address 127.3.85.129:34359)
0619 18:44:14.347213 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.347229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:14.347416 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:14.347813 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:14.347817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:14.347842 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:14.348301 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.348313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.349399 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.349404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.351348 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:14.351365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.352688 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.352696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:14.352869 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.352893 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:44:14.353010 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:44:14.353160 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":109,"thread_start_us":1204,"threads_started":1}
I20260619 18:44:14.355266 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1a9956877b4953933a12862cfe5b84" candidate_uuid: "02755be784b14113ac0e1c8842ed7314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" is_pre_election: true
I20260619 18:44:14.355268 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1a9956877b4953933a12862cfe5b84" candidate_uuid: "02755be784b14113ac0e1c8842ed7314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29b83535f40417685086498c3263609" is_pre_election: true
I20260619 18:44:14.355902 18117 raft_consensus.cc:2454] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02755be784b14113ac0e1c8842ed7314 in term 0.
I20260619 18:44:14.355902 18248 raft_consensus.cc:2454] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02755be784b14113ac0e1c8842ed7314 in term 0.
I20260619 18:44:14.356755 17920 leader_election.cc:302] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02755be784b14113ac0e1c8842ed7314, 39a89ebb2c714ea8a2bea890bd5e897a, c29b83535f40417685086498c3263609; no voters:
I20260619 18:44:14.357298 18315 raft_consensus.cc:2781] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:14.357430 18315 raft_consensus.cc:482] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:14.357569 18315 raft_consensus.cc:3037] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:14.360021 18315 raft_consensus.cc:504] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.360790 18315 leader_election.cc:288] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [CANDIDATE]: Term 1 election: Requested vote from peers c29b83535f40417685086498c3263609 (127.3.85.130:40109), 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745)
I20260619 18:44:14.361465 18117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1a9956877b4953933a12862cfe5b84" candidate_uuid: "02755be784b14113ac0e1c8842ed7314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29b83535f40417685086498c3263609"
I20260619 18:44:14.361464 18248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1a9956877b4953933a12862cfe5b84" candidate_uuid: "02755be784b14113ac0e1c8842ed7314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
I20260619 18:44:14.361717 18248 raft_consensus.cc:3037] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:14.361721 18117 raft_consensus.cc:3037] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:14.365381 18248 raft_consensus.cc:2454] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02755be784b14113ac0e1c8842ed7314 in term 1.
I20260619 18:44:14.365381 18117 raft_consensus.cc:2454] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02755be784b14113ac0e1c8842ed7314 in term 1.
I20260619 18:44:14.365938 17920 leader_election.cc:302] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02755be784b14113ac0e1c8842ed7314, 39a89ebb2c714ea8a2bea890bd5e897a, c29b83535f40417685086498c3263609; no voters:
I20260619 18:44:14.366334 18315 raft_consensus.cc:2781] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:14.366858 18315 raft_consensus.cc:686] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 1 LEADER]: Becoming Leader. State: Replica: 02755be784b14113ac0e1c8842ed7314, State: Running, Role: LEADER
I20260619 18:44:14.367478 18315 consensus_queue.cc:237] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } }
I20260619 18:44:14.372951 17844 catalog_manager.cc:5495] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02755be784b14113ac0e1c8842ed7314 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "02755be784b14113ac0e1c8842ed7314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02755be784b14113ac0e1c8842ed7314" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39157 } health_report { overall_health: HEALTHY } } }
I20260619 18:44:14.408026 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.402100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.3.85.129:39157)
0619 18:44:14.402336 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.402344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.402371 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:44:14.402546 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.402550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.402644 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.402803 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:14.403197 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.403999 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.405451 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.405883 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.405986 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.406660 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:14.407240 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:14.407382 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.407698 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.407805 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:44:14.411010 18294 tablet.cc:2369] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:14.417976 18117 raft_consensus.cc:1261] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Refusing update from remote peer 02755be784b14113ac0e1c8842ed7314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:14.417982 18248 raft_consensus.cc:1261] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Refusing update from remote peer 02755be784b14113ac0e1c8842ed7314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:14.419222 18320 consensus_queue.cc:1035] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c29b83535f40417685086498c3263609" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:14.419624 18315 consensus_queue.cc:1035] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a89ebb2c714ea8a2bea890bd5e897a" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:44:14.434059 18163 tablet.cc:2369] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:14.449694 17844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:14.450030 17844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:14.454223 17844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Sending DeleteTablet for 3 replicas of tablet de1a9956877b4953933a12862cfe5b84
I20260619 18:44:14.455901 18097 tablet_service.cc:1514] Processing DeleteTablet for tablet de1a9956877b4953933a12862cfe5b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:14 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35916
I20260619 18:44:14.456076 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17882
I20260619 18:44:14.456065 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet de1a9956877b4953933a12862cfe5b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:14 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35628
I20260619 18:44:14.457130 18333 tablet_replica.cc:330] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: stopping tablet replica
I20260619 18:44:14.457137 18228 tablet_service.cc:1514] Processing DeleteTablet for tablet de1a9956877b4953933a12862cfe5b84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:14 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:35206
I20260619 18:44:14.457643 18333 raft_consensus.cc:2229] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:14.458195 18333 raft_consensus.cc:2258] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:14.459307 18334 tablet_replica.cc:330] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: stopping tablet replica
I20260619 18:44:14.459790 18334 raft_consensus.cc:2229] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:14.460582 18334 raft_consensus.cc:2258] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:14.461443 18333 ts_tablet_manager.cc:1905] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:14.466960 18334 ts_tablet_manager.cc:1905] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:14.474212 18333 ts_tablet_manager.cc:1918] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:14.474555 18333 log.cc:1196] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/de1a9956877b4953933a12862cfe5b84
I20260619 18:44:14.475279 18333 ts_tablet_manager.cc:1939] T de1a9956877b4953933a12862cfe5b84 P c29b83535f40417685086498c3263609: Deleting consensus metadata
I20260619 18:44:14.476576 18335 tablet_replica.cc:330] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: stopping tablet replica
I20260619 18:44:14.477285 18335 raft_consensus.cc:2229] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:14.477499 17829 catalog_manager.cc:4841] TS c29b83535f40417685086498c3263609 (127.3.85.130:40109): tablet de1a9956877b4953933a12862cfe5b84 (table test-table [id=d2e17bdf07ec413898e623bbde53bb52]) successfully deleted
I20260619 18:44:14.477918 18335 raft_consensus.cc:2258] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:14.481412 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.459574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.3.85.129:39157)
0619 18:44:14.461260 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.461269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.461296 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:44:14.461369 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.461374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.461459 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.461635 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:14.463659 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.465230 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.467276 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.468309 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.468494 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.472091 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:14.472115 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:14.472142 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:14.472233 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:14.477405 (+ 5172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.478440 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.478486 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.478491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.478703 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.479333 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.479342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.479345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.479902 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:14.480260 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.480653 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.480963 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1485}
I20260619 18:44:14.482529 18025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:14.482712 18335 ts_tablet_manager.cc:1905] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:14.490873 18335 ts_tablet_manager.cc:1918] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:14.491096 18335 log.cc:1196] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/de1a9956877b4953933a12862cfe5b84
I20260619 18:44:14.491683 18335 ts_tablet_manager.cc:1939] T de1a9956877b4953933a12862cfe5b84 P 39a89ebb2c714ea8a2bea890bd5e897a: Deleting consensus metadata
I20260619 18:44:14.493808 17831 catalog_manager.cc:4841] TS 39a89ebb2c714ea8a2bea890bd5e897a (127.3.85.131:33745): tablet de1a9956877b4953933a12862cfe5b84 (table test-table [id=d2e17bdf07ec413898e623bbde53bb52]) successfully deleted
I20260619 18:44:14.582027 18334 ts_tablet_manager.cc:1918] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:14.582363 18334 log.cc:1196] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/de1a9956877b4953933a12862cfe5b84
I20260619 18:44:14.583112 18334 ts_tablet_manager.cc:1939] T de1a9956877b4953933a12862cfe5b84 P 02755be784b14113ac0e1c8842ed7314: Deleting consensus metadata
I20260619 18:44:14.583739 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17882
I20260619 18:44:14.603106 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18034
W20260619 18:44:14.603120 17829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:39157 (error 108)
I20260619 18:44:14.605214 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.604769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35660)
0619 18:44:14.604958 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.605063 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 18:44:14.605674 17829 catalog_manager.cc:4584] TS 02755be784b14113ac0e1c8842ed7314 (127.3.85.129:39157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de1a9956877b4953933a12862cfe5b84: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
I20260619 18:44:14.618665 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.604008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.3.85.130:40109)
0619 18:44:14.604215 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.604224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.604330 (+ 106us) server_negotiation.cc:407] Connection header received
0619 18:44:14.604491 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.604498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.604593 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.604708 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:14.606656 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.607402 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.608915 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.609351 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.609518 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.612433 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:14.612450 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:14.612466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:14.612508 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:14.615498 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.616381 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.616390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.616393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.616528 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.616942 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.616947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.616951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.617347 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:14.617690 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.617998 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.618238 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:44:14.619500 18157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:14.659641 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.659018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35662)
0619 18:44:14.659279 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.659425 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:44:14.660840 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.660445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35668)
0619 18:44:14.660603 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.660712 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:44:14.715631 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18034
I20260619 18:44:14.722245 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.721618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35678)
0619 18:44:14.721867 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.721997 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 18:44:14.723742 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.723229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35692)
0619 18:44:14.723439 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.723550 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:44:14.736033 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18165
I20260619 18:44:14.753454 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.736980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.3.85.131:33745)
0619 18:44:14.737208 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:14.737215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:14.737311 (+ 96us) server_negotiation.cc:407] Connection header received
0619 18:44:14.737567 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:14.737572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:14.737647 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:14.737769 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:14.739417 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.740469 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.742124 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:14.743006 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:14.743178 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:14.746142 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:14.746159 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:14.746176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:14.746237 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:14.749870 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.750904 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.750913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.750915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.751098 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:14.751542 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:14.751547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:14.751549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:14.751947 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:14.752445 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:14.752817 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 18:44:14.753123 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:44:14.754544 18288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:14.804381 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.803789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35700)
0619 18:44:14.804049 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.804184 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 18:44:14.805576 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.805071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35714)
0619 18:44:14.805331 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.805426 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 18:44:14.848737 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18165
I20260619 18:44:14.865772 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 17812
I20260619 18:44:14.867038 17875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:14.988171 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:14.987393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39157 (local address 127.0.0.1:35722)
0619 18:44:14.987713 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:14.987861 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 18:44:14.988387 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 17812
I20260619 18:44:15.011289 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--webserver_interface=127.3.85.190
--webserver_port=42733
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:15.366081 18341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:15.366425 18341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:15.366616 18341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:15.374328 18341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:15.374542 18341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:15.374657 18341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:15.374771 18341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:15.374858 18341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:15.385227 18341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=42733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:15.386716 18341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:15.388113 18341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:15.401302 18347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:15.401332 18346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:15.401391 18349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:15.402263 18341 server_base.cc:1034] running on GCE node
I20260619 18:44:15.403240 18341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:15.404879 18341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:15.406152 18341 hybrid_clock.cc:648] HybridClock initialized: now 1781894655406075 us; error 67 us; skew 500 ppm
I20260619 18:44:15.406617 18341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:15 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:15.410364 18341 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:15.412186 18341 webserver.cc:460] Webserver started at http://127.3.85.190:42733/ using document root <none> and password file <none>
I20260619 18:44:15.413050 18341 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:15.413239 18341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:15.419706 18341 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:44:15.423197 18356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:15.424921 18341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:44:15.425194 18341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "afc045bcdf93480187bb2054ac9652c1"
format_stamp: "Formatted at 2026-06-19 18:44:09 on dist-test-slave-09fp"
server_key: "df9cd311e869518cb50beeac966f36db"
server_key_iv: "d13e5887a836c3ad0418151ff747f709"
server_key_version: "encryptionkey@0"
I20260619 18:44:15.426337 18341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:15.444429 18341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:15.449496 18341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:15.449810 18341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:15.471184 18341 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:46203
I20260619 18:44:15.471222 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:46203 every 8 connection(s)
I20260619 18:44:15.472851 18341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:15.476699 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18341
I20260619 18:44:15.477764 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:39157
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=41779
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:15.481966 18408 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:44:15.487496 18408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap starting.
I20260619 18:44:15.509750 18408 log.cc:824] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:15.527980 18408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:44:15.528621 18408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1: Bootstrap complete.
I20260619 18:44:15.538786 18408 raft_consensus.cc:348] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:15.539773 18408 raft_consensus.cc:729] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Initialized, Role: FOLLOWER
I20260619 18:44:15.540597 18408 consensus_queue.cc:260] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:15.540877 18408 raft_consensus.cc:388] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:15.541002 18408 raft_consensus.cc:482] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:15.541208 18408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:44:15.545476 18408 raft_consensus.cc:504] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:15.546139 18408 leader_election.cc:302] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afc045bcdf93480187bb2054ac9652c1; no voters:
I20260619 18:44:15.546941 18408 leader_election.cc:288] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:44:15.547108 18412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:44:15.548501 18412 raft_consensus.cc:686] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [term 3 LEADER]: Becoming Leader. State: Replica: afc045bcdf93480187bb2054ac9652c1, State: Running, Role: LEADER
I20260619 18:44:15.549247 18412 consensus_queue.cc:237] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } }
I20260619 18:44:15.549583 18408 sys_catalog.cc:564] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:15.552704 18414 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afc045bcdf93480187bb2054ac9652c1. Latest consensus state: current_term: 3 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:15.552716 18413 sys_catalog.cc:455] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "afc045bcdf93480187bb2054ac9652c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afc045bcdf93480187bb2054ac9652c1" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 46203 } } }
I20260619 18:44:15.553267 18413 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:15.553262 18414 sys_catalog.cc:458] T 00000000000000000000000000000000 P afc045bcdf93480187bb2054ac9652c1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:15.565500 18420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:15.575860 18420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:15.577702 18420 catalog_manager.cc:1259] Loaded cluster ID: 2589391bb5984c47957ac33edf1bdbad
I20260619 18:44:15.577893 18420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 18:44:15.580087 18420 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c1ce7e9ad google::LogMessage::Fail() at ??:0
@ 0x7f9c1ce8188a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c1ce7e4de google::LogMessage::Flush() at ??:0
@ 0x7f9c1ce7fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c328b1512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9c328ae976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c32936610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9c32936439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c1faca345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c1facd91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9c1facd799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c1faa8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c22d826db start_thread at ??:0
@ 0x7f9c1bd1861f clone at ??:0
W20260619 18:44:15.915833 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:15.916167 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:15.916280 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:15.923856 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:15.923998 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:15.924117 18410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:44:15.934355 18410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:39157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=41779
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:15.935747 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:15.937158 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:15.952044 18450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:15.952070 18449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:15.951975 18452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:15.956550 18410 server_base.cc:1034] running on GCE node
I20260619 18:44:15.957320 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:15.958539 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:15.959815 18410 hybrid_clock.cc:648] HybridClock initialized: now 1781894655959753 us; error 57 us; skew 500 ppm
I20260619 18:44:15.960230 18410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:15 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:15.964239 18410 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:44:15.966048 18410 webserver.cc:460] Webserver started at http://127.3.85.129:41779/ using document root <none> and password file <none>
I20260619 18:44:15.966946 18410 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:15.967101 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:15.973973 18410 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 18:44:15.977608 18459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:15.979393 18410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 18:44:15.979611 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "02755be784b14113ac0e1c8842ed7314"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "9f0e0573e00500a706931441c4890464"
server_key_iv: "e73373302aff60eb411886a45b159359"
server_key_version: "encryptionkey@0"
I20260619 18:44:15.980727 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:15.998369 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:16.003290 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:16.003636 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:16.004755 18410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:16.006283 18410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:16.006386 18410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.006583 18410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:16.006695 18410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.042979 18410 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:39157
I20260619 18:44:16.043020 18571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:39157 every 8 connection(s)
I20260619 18:44:16.044658 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:16.047991 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18410
I20260619 18:44:16.048692 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:40109
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=35149
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:16.050429 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.048094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:54269)
0619 18:44:16.049092 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.049379 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":426,"threads_started":1}
W20260619 18:44:16.051730 18572 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:46203 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:46203: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
I20260619 18:44:16.052820 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.052405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:45509)
0619 18:44:16.052587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.052689 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:44:16.053948 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.053427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:54683)
0619 18:44:16.053648 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.053768 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:44:16.054309 18572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:44:16.399732 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:16.400056 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:16.400209 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:16.407898 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:16.408041 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:16.408232 18575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:44:16.419281 18575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:40109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=35149
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:16.420707 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:16.422143 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:16.436112 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:16.436072 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:16.436153 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:16.436967 18575 server_base.cc:1034] running on GCE node
I20260619 18:44:16.437686 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:16.438844 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:16.440095 18575 hybrid_clock.cc:648] HybridClock initialized: now 1781894656440048 us; error 48 us; skew 500 ppm
I20260619 18:44:16.440451 18575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:16 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894656, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:16.443898 18575 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:44:16.445678 18575 webserver.cc:460] Webserver started at http://127.3.85.130:35149/ using document root <none> and password file <none>
I20260619 18:44:16.446798 18575 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:16.446930 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:16.453204 18575 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:44:16.456645 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.458269 18575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:44:16.458541 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "c29b83535f40417685086498c3263609"
format_stamp: "Formatted at 2026-06-19 18:44:10 on dist-test-slave-09fp"
server_key: "c10de420e82cf32da6f03194e8067d5b"
server_key_iv: "aa6fefe76d62596bc95ebf9ec670a735"
server_key_version: "encryptionkey@0"
I20260619 18:44:16.459792 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:16.485051 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:16.490978 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:16.491300 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:16.492445 18575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:16.493933 18575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:16.494027 18575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.494203 18575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:16.494292 18575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.530128 18575 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40109
I20260619 18:44:16.530272 18702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40109 every 8 connection(s)
I20260619 18:44:16.531797 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:16.535871 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18575
I20260619 18:44:16.536666 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:33745
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:46203
--webserver_port=33133
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:37291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:16.537824 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.535014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:47195)
0619 18:44:16.536045 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.536334 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":383,"threads_started":1}
W20260619 18:44:16.538594 18703 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:46203 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:46203: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
I20260619 18:44:16.539671 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.539183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:57677)
0619 18:44:16.539398 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.539504 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 18:44:16.540796 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:16.540317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:52435)
0619 18:44:16.540507 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:16.540615 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 18:44:16.541107 18703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:44:16.892498 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:16.892823 18706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:16.893002 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:16.900991 18706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:16.901136 18706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:16.901362 18706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:44:16.912503 18706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:33745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=33133
--tserver_master_addrs=127.3.85.190:46203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:16.913969 18706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:16.915441 18706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:16.930471 18712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:16.930471 18711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:16.930501 18714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:16.932111 18706 server_base.cc:1034] running on GCE node
I20260619 18:44:16.933218 18706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:16.934420 18706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:16.935770 18706 hybrid_clock.cc:648] HybridClock initialized: now 1781894656935713 us; error 41 us; skew 500 ppm
I20260619 18:44:16.936192 18706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:16 dist-test-slave-09fp krb5kdc[17259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894656, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:16.940586 18706 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:44:16.942433 18706 webserver.cc:460] Webserver started at http://127.3.85.131:33133/ using document root <none> and password file <none>
I20260619 18:44:16.943323 18706 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:16.943444 18706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:16.949748 18706 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:44:16.953672 18721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.956219 18706 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 18:44:16.956583 18706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "39a89ebb2c714ea8a2bea890bd5e897a"
format_stamp: "Formatted at 2026-06-19 18:44:11 on dist-test-slave-09fp"
server_key: "5ac2e61de2d13700e6535d0011454d13"
server_key_iv: "29c0c5485644fa16ba6a9b121e0b7d5c"
server_key_version: "encryptionkey@0"
I20260619 18:44:16.957711 18706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:16.984093 18706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:16.990168 18706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:16.990566 18706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:16.991725 18706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:16.993427 18706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:16.993532 18706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:16.993613 18706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:16.993669 18706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:17.032245 18706 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:33745
I20260619 18:44:17.032313 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:33745 every 8 connection(s)
I20260619 18:44:17.034025 18706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:17.039472 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:17.037292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:51229)
0619 18:44:17.038135 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:17.038392 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":344,"threads_started":1}
W20260619 18:44:17.040419 18834 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:46203 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:46203: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
I20260619 18:44:17.041494 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:17.041095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:42173)
0619 18:44:17.041274 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:17.041367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:44:17.042886 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:17.042500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:55497)
0619 18:44:17.042649 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:17.042743 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:44:17.043267 18834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:44:17.044414 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18706
I20260619 18:44:17.056061 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:17.055094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:60897)
0619 18:44:17.055683 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:17.055813 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":316,"threads_started":1}
I20260619 18:44:17.542912 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:17.541920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:51985)
0619 18:44:17.542518 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:17.542671 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
I20260619 18:44:18.044926 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:18.043929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:44761)
0619 18:44:18.044502 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:18.044624 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260619 18:44:18.058018 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:18.057171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:45357)
0619 18:44:18.057756 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:18.057846 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":307,"threads_started":1}
I20260619 18:44:18.544729 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:18.543885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:59555)
0619 18:44:18.544449 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:18.544539 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":335,"threads_started":1}
I20260619 18:44:19.046975 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:19.046162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:53987)
0619 18:44:19.046708 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:19.046806 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":289,"threads_started":1}
I20260619 18:44:19.060195 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:19.059277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:58919)
0619 18:44:19.059888 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:19.059999 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":371,"threads_started":1}
I20260619 18:44:19.546756 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:19.545792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:37661)
0619 18:44:19.546414 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:19.546549 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":265,"threads_started":1}
I20260619 18:44:20.048942 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:20.048040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:39713)
0619 18:44:20.048632 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:20.048746 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
I20260619 18:44:20.062222 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:20.061327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:55833)
0619 18:44:20.061833 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:20.061917 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":237,"threads_started":1}
I20260619 18:44:20.548609 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:20.547851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:41235)
0619 18:44:20.548341 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:20.548433 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260619 18:44:21.051043 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:21.050096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:43547)
0619 18:44:21.050722 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:21.050842 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":352,"threads_started":1}
I20260619 18:44:21.064307 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:21.063261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:51819)
0619 18:44:21.063919 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:21.064057 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":313,"threads_started":1}
I20260619 18:44:21.551044 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:21.549848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:57743)
0619 18:44:21.550542 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:21.550650 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":291,"threads_started":1}
I20260619 18:44:22.052917 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:22.052131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:32781)
0619 18:44:22.052641 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:22.052733 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":271,"threads_started":1}
I20260619 18:44:22.066190 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:22.065298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:52253)
0619 18:44:22.065866 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:22.066012 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":323,"threads_started":1}
I20260619 18:44:22.553113 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:22.552114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:51411)
0619 18:44:22.552825 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:22.552940 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":272,"threads_started":1}
I20260619 18:44:23.055006 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:23.054113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:41551)
0619 18:44:23.054729 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:23.054830 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":358,"threads_started":1}
I20260619 18:44:23.068085 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:23.067237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:34135)
0619 18:44:23.067764 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:23.067857 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":236,"threads_started":1}
I20260619 18:44:23.555606 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:23.554325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:36497)
0619 18:44:23.555051 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:23.555160 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":352,"threads_started":1}
I20260619 18:44:24.057041 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:24.056091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:41237)
0619 18:44:24.056725 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:24.056822 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":296,"threads_started":1}
I20260619 18:44:24.069991 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:24.069236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:40595)
0619 18:44:24.069716 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:24.069819 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260619 18:44:24.557629 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:24.556850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:47267)
0619 18:44:24.557342 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:24.557431 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260619 18:44:25.059083 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:25.058275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:49573)
0619 18:44:25.058817 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:25.058920 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
I20260619 18:44:25.071990 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:25.071151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:36691)
0619 18:44:25.071713 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:25.071813 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260619 18:44:25.559616 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:25.558754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:48001)
0619 18:44:25.559301 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:25.559431 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":264,"threads_started":1}
I20260619 18:44:26.060916 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:26.060146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:39587)
0619 18:44:26.060614 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:26.060708 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":208,"threads_started":1}
I20260619 18:44:26.073791 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:26.073055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:35165)
0619 18:44:26.073534 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:26.073624 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
I20260619 18:44:26.561709 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:26.560811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:48645)
0619 18:44:26.561368 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:26.561468 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":281,"threads_started":1}
I20260619 18:44:27.062893 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:27.061983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:32879)
0619 18:44:27.062633 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:27.062728 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":348,"threads_started":1}
I20260619 18:44:27.075896 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:27.074996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:55259)
0619 18:44:27.075566 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:27.075667 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260619 18:44:27.564016 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:27.563190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:38013)
0619 18:44:27.563732 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:27.563833 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
I20260619 18:44:28.064796 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:28.063894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:33309)
0619 18:44:28.064419 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:28.064581 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":271,"threads_started":1}
I20260619 18:44:28.077988 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:28.077086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:38895)
0619 18:44:28.077624 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:28.077716 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":307,"threads_started":1}
I20260619 18:44:28.566107 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:28.565263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:52535)
0619 18:44:28.565846 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:28.565941 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260619 18:44:29.066926 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:29.065814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:44395)
0619 18:44:29.066651 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:29.066762 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":481,"threads_started":1}
I20260619 18:44:29.080128 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:29.079172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:36889)
0619 18:44:29.079794 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:29.079922 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":345,"threads_started":1}
I20260619 18:44:29.567838 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:29.567049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:58531)
0619 18:44:29.567531 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:29.567628 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":249,"threads_started":1}
I20260619 18:44:30.068717 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:30.067819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:53599)
0619 18:44:30.068382 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:30.068482 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":220,"threads_started":1}
I20260619 18:44:30.081898 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:30.081148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:45249)
0619 18:44:30.081615 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:30.081705 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260619 18:44:30.569643 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:30.568819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:59431)
0619 18:44:30.569363 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:30.569455 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":295,"threads_started":1}
I20260619 18:44:31.070744 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:31.069748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:33787)
0619 18:44:31.070406 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:31.070557 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":404,"threads_started":1}
I20260619 18:44:31.083977 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:31.083221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.129:40471)
0619 18:44:31.083684 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:31.083795 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260619 18:44:31.571660 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:31.570728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.130:41259)
0619 18:44:31.571258 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:31.571397 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":261,"threads_started":1}
I20260619 18:44:32.049897 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18410
I20260619 18:44:32.072083 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:32.051352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.3.85.129:39157)
0619 18:44:32.052091 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:32.052118 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:32.052173 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:44:32.054331 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:32.054344 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:32.054940 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:32.055297 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:32.056922 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.058324 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.059866 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.060706 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.060874 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:32.064133 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:32.064145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:32.064165 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:32.064216 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:32.068100 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.069044 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.069053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.069060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.069273 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.069773 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.069777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.069779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.070430 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:32.070769 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:32.071215 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 18:44:32.071690 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":324,"threads_started":1}
I20260619 18:44:32.072808 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:32.071989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:46203 (local address 127.3.85.131:59207)
0619 18:44:32.072482 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:32.072630 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:46203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":447,"threads_started":1}
I20260619 18:44:32.073751 18567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:32.162438 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18410
I20260619 18:44:32.181231 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18575
I20260619 18:44:32.200788 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:32.182233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.3.85.130:40109)
0619 18:44:32.182990 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:32.183004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:32.183043 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:44:32.184660 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:32.184671 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:32.185137 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:32.185541 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:32.187121 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.188393 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.189694 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.190507 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.190700 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:32.193527 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:32.193537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:32.193555 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:32.193604 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:32.196814 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.197713 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.197721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.197729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.197936 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.198413 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.198416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.198419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.199099 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:32.199351 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:32.199805 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 18:44:32.200480 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":492,"threads_started":1}
I20260619 18:44:32.202381 18698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:32.289667 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18575
I20260619 18:44:32.308838 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18706
I20260619 18:44:32.328800 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:32.309980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.3.85.131:33745)
0619 18:44:32.310680 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:32.310695 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:32.310740 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:44:32.312476 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:32.312488 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:32.312960 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:32.313358 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:32.314993 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.316276 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.317712 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:32.318530 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:32.318757 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:32.321595 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:32.321604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:32.321623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:32.321677 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:32.324985 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.325843 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.325852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.325865 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.326059 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:32.326534 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:32.326541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:32.326545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:32.327153 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:32.327519 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:32.328019 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 18:44:32.328490 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":300,"threads_started":1}
I20260619 18:44:32.330507 18829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:32.418026 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18706
2026-06-19T18:44:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24781 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:44:32 dist-test-slave-09fp krb5kdc[18892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:44:32 dist-test-slave-09fp krb5kdc[18892](info): set up 2 sockets
Jun 19 18:44:32 dist-test-slave-09fp krb5kdc[18892](info): commencing operation
krb5kdc: starting...
W20260619 18:44:33.531627 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:44:33 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:44:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:44:33.795410 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:34.148195 18905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:34.148524 18905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:34.148717 18905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:34.156430 18905 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:44:34.156556 18905 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:44:34.156602 18905 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:44:34.156692 18905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:34.156771 18905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:34.156894 18905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:34.156962 18905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:34.157012 18905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:34.167464 18905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:34.168896 18905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:34.170333 18905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:34.183619 18913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:34.183696 18911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:34.183681 18910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:34.184495 18905 server_base.cc:1034] running on GCE node
I20260619 18:44:34.185390 18905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:34.187040 18905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:34.188329 18905 hybrid_clock.cc:648] HybridClock initialized: now 1781894674188254 us; error 53 us; skew 500 ppm
I20260619 18:44:34.188715 18905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:34 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:34.192420 18905 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:34.194128 18905 webserver.cc:460] Webserver started at http://127.3.85.190:37419/ using document root <none> and password file <none>
I20260619 18:44:34.194994 18905 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:34.195153 18905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:34.195530 18905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:34.199016 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "a2a7ba4faaa9417f92aca3c24be5294a"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "f761b00ff307534b89e826f66d168274"
server_key_iv: "720d5399cc86351e8f6716accb4515ac"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.200004 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "a2a7ba4faaa9417f92aca3c24be5294a"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "f761b00ff307534b89e826f66d168274"
server_key_iv: "720d5399cc86351e8f6716accb4515ac"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.206286 18905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260619 18:44:34.211117 18920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:34.212849 18905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:44:34.213109 18905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "a2a7ba4faaa9417f92aca3c24be5294a"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "f761b00ff307534b89e826f66d168274"
server_key_iv: "720d5399cc86351e8f6716accb4515ac"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.213388 18905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:34.229369 18905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:34.234702 18905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:34.235046 18905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:34.256201 18905 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:32829
I20260619 18:44:34.256222 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:32829 every 8 connection(s)
I20260619 18:44:34.257810 18905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:34.261847 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18905
I20260619 18:44:34.262109 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:44:34.262710 18972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:34.262887 3414 external_mini_cluster.cc:1428] Setting key dd4b9a25d92d7961a3c20cdc473ca85e
I20260619 18:44:34.279810 18972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap starting.
Jun 19 18:44:34 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:34.284363 18972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:34.285643 18972 log.cc:824] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:34.288861 18972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: No bootstrap required, opened a new log
I20260619 18:44:34.290364 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:34.265315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.3.85.190:32829)
0619 18:44:34.266520 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:34.266534 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:34.266585 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:44:34.268369 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:34.268395 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:34.268880 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:34.269310 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:34.271391 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.272766 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.274903 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.275726 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.275892 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:34.281850 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:34.281860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:34.281876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:34.281936 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:34.285854 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:34.286752 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:34.286763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:34.286770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:34.286958 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:34.287366 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:34.287370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:34.287373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:34.287949 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:34.288291 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:34.288760 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 18:44:34.289300 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":622,"threads_started":1}
I20260619 18:44:34.298056 18972 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:34.298591 18972 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:34.298723 18972 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Initialized, Role: FOLLOWER
I20260619 18:44:34.300724 18972 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:34.301548 18972 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:34.302124 18972 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:34.302743 18972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:34.307958 18972 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:34.309176 18972 leader_election.cc:302] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a7ba4faaa9417f92aca3c24be5294a; no voters:
I20260619 18:44:34.309902 18972 leader_election.cc:288] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:44:34.310096 18977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:34.310931 18977 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 LEADER]: Becoming Leader. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Running, Role: LEADER
I20260619 18:44:34.311602 18977 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:34.312363 18972 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:34.316133 18978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:44:34.316133 18979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a7ba4faaa9417f92aca3c24be5294a. Latest consensus state: current_term: 1 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:44:34.316604 18978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:34.316642 18979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:34.322198 18985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:34.328118 18985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:34.339071 18985 catalog_manager.cc:1347] Generated new cluster ID: b2774c131d89449d9a20382c8aea7254
I20260619 18:44:34.339252 18985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:34.356276 18985 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:44:34.357772 18985 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:34.375901 18985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Generated new TSK 0
I20260619 18:44:34.376919 18985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:44:34.451293 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:34.854923 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:34.855239 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:34.855342 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:34.863088 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:34.863220 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:34.863340 18998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:44:34.873550 18998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:34.874959 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:34.876304 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:34.890748 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:34.890725 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:34.890645 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:34.891722 18998 server_base.cc:1034] running on GCE node
I20260619 18:44:34.892495 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:34.893599 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:34.894850 18998 hybrid_clock.cc:648] HybridClock initialized: now 1781894674894772 us; error 76 us; skew 500 ppm
I20260619 18:44:34.895226 18998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:34 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:34.898878 18998 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:44:34.900679 18998 webserver.cc:460] Webserver started at http://127.3.85.129:46507/ using document root <none> and password file <none>
I20260619 18:44:34.901517 18998 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:34.901690 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:34.902030 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:34.905273 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "9d0f1e0a2fe1465084bc3926b4049421"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "d164c6e08e1f5e93a2fc2e56482794cb"
server_key_iv: "7d9a16d84633a5f960b9499e012dc40b"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.906245 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "9d0f1e0a2fe1465084bc3926b4049421"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "d164c6e08e1f5e93a2fc2e56482794cb"
server_key_iv: "7d9a16d84633a5f960b9499e012dc40b"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.911995 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 18:44:34.916057 19013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:34.917670 18998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:44:34.917914 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "9d0f1e0a2fe1465084bc3926b4049421"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "d164c6e08e1f5e93a2fc2e56482794cb"
server_key_iv: "7d9a16d84633a5f960b9499e012dc40b"
server_key_version: "encryptionkey@0"
I20260619 18:44:34.918180 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:34.930629 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:34.935599 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:34.935954 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:34.937047 18998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:34.938685 18998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:34.938810 18998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:34.938970 18998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:34.939046 18998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:34.977396 18998 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:42503
I20260619 18:44:34.977406 19125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:42503 every 8 connection(s)
I20260619 18:44:34.979180 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:34.983403 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 18998
I20260619 18:44:34.983696 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:44:34.984591 3414 external_mini_cluster.cc:1428] Setting key fb4eeccaa43574b988d6047c620dbee1
Jun 19 18:44:34 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894674, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:35.003162 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:34.982306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:42993 (local address 127.3.85.190:32829)
0619 18:44:34.982918 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:34.982927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:34.984233 (+ 1306us) server_negotiation.cc:407] Connection header received
0619 18:44:34.986820 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:34.986825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:34.986890 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:34.987029 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:34.988839 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.990079 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.991932 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.992687 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.992851 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:34.996789 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:34.996795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:34.996798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:34.996837 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:34.999657 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:35.000576 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:35.000587 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:35.000589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:35.000726 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:35.001225 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:35.001238 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:35.001241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:35.001529 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:35.001700 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:35.002503 (+ 803us) server_negotiation.cc:299] Negotiation successful
0619 18:44:35.002766 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":295,"threads_started":1}
I20260619 18:44:35.004206 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:34.982985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:42993)
0619 18:44:34.984007 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:34.984064 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:34.986317 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:34.987319 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:34.987329 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:34.987937 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:34.988605 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:34.988628 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.990261 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.990266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:34.991734 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:34.991754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:34.992941 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:34.992946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:34.993173 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:34.993972 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:34.993998 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:34.996483 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:34.999862 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:34.999873 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:34.999896 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:35.000410 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:35.000858 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:35.000863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:35.000867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:35.001086 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:35.001674 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:35.001682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:35.002068 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 18:44:35.002356 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 18:44:35.002981 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":373,"threads_started":1}
I20260619 18:44:35.006062 19126 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:35.006521 19126 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:35.007530 19126 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:44:35.010268 18937 ts_manager.cc:194] Registered new tserver with Master: 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503)
I20260619 18:44:35.012199 18937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:42993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:44:35.042912 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:35.393253 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:35.393677 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:35.393864 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:35.401808 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:35.402007 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:35.402208 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:44:35.412817 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:35.414372 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:35.415884 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:35.430876 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:35.430954 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:35.430894 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:35.431643 19132 server_base.cc:1034] running on GCE node
I20260619 18:44:35.432586 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:35.433658 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:35.434872 19132 hybrid_clock.cc:648] HybridClock initialized: now 1781894675434824 us; error 40 us; skew 500 ppm
I20260619 18:44:35.435242 19132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:35 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:35.438753 19132 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:44:35.440470 19132 webserver.cc:460] Webserver started at http://127.3.85.130:37169/ using document root <none> and password file <none>
I20260619 18:44:35.441275 19132 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:35.441434 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:35.441776 19132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:35.445017 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "98a949bde1714d96b2a156b689557992"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "1cdad702017f2c294ef99440b2e3bc1e"
server_key_iv: "07d08d16541aec0c9a67f965e89decf5"
server_key_version: "encryptionkey@0"
I20260619 18:44:35.445888 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "98a949bde1714d96b2a156b689557992"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "1cdad702017f2c294ef99440b2e3bc1e"
server_key_iv: "07d08d16541aec0c9a67f965e89decf5"
server_key_version: "encryptionkey@0"
I20260619 18:44:35.451884 19132 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:44:35.455972 19147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:35.457700 19132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:44:35.457959 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "98a949bde1714d96b2a156b689557992"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "1cdad702017f2c294ef99440b2e3bc1e"
server_key_iv: "07d08d16541aec0c9a67f965e89decf5"
server_key_version: "encryptionkey@0"
I20260619 18:44:35.458231 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:35.483436 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:35.488202 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:35.488536 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:35.489596 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:35.491133 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:35.491255 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:35.491410 19132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:35.491487 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:35.528553 19132 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:33733
I20260619 18:44:35.528612 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:33733 every 8 connection(s)
I20260619 18:44:35.529943 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:35.539305 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19132
I20260619 18:44:35.539506 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:44:35.540205 3414 external_mini_cluster.cc:1428] Setting key 36f0fd282b55060364d3be6a98c99634
Jun 19 18:44:35 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:35.556639 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:35.532735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:43267 (local address 127.3.85.190:32829)
0619 18:44:35.533193 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:35.533199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:35.534330 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 18:44:35.536308 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:35.536313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:35.536391 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:35.536497 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:35.538286 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:35.539528 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:35.542388 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:35.543050 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:35.543161 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:35.547994 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:35.548004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:35.548013 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:35.548068 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:35.551961 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:35.553373 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:35.553383 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:35.553390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:35.553601 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:35.554150 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:35.554159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:35.554165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:35.554656 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:35.554826 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:35.555744 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 18:44:35.556073 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260619 18:44:35.557359 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:35.533388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:43267)
0619 18:44:35.534146 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:35.534193 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:35.535991 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:35.536769 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:35.536778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:35.537368 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:35.538042 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:35.538063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:35.539860 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:35.539868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:35.542169 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:35.542189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:35.543326 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:35.543335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:35.543729 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:35.544542 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:35.544586 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:35.547664 (+ 3078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:35.552200 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:35.552225 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:35.552250 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:35.553134 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:35.553761 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:35.553765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:35.553768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:35.554013 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:35.554831 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:35.554839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:35.555277 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 18:44:35.555578 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 18:44:35.556054 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
I20260619 18:44:35.559846 19260 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:35.560267 19260 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:35.561080 19260 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
I20260619 18:44:35.562983 18937 ts_manager.cc:194] Registered new tserver with Master: 98a949bde1714d96b2a156b689557992 (127.3.85.130:33733)
I20260619 18:44:35.563966 18937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:43267
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:44:35.603469 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:35.949141 19266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:35.949534 19266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:35.949723 19266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:35.957566 19266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:35.957763 19266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:35.957965 19266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:44:35.968289 19266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:35.969712 19266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:35.971122 19266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:35.987334 19271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:35.987334 19272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:35.987353 19274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:35.989018 19266 server_base.cc:1034] running on GCE node
I20260619 18:44:35.989791 19266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:35.991045 19266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:35.992368 19266 hybrid_clock.cc:648] HybridClock initialized: now 1781894675992308 us; error 56 us; skew 500 ppm
I20260619 18:44:35.992751 19266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:35 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:35.996631 19266 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:44:35.998737 19266 webserver.cc:460] Webserver started at http://127.3.85.131:36701/ using document root <none> and password file <none>
I20260619 18:44:35.999580 19266 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:35.999820 19266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:36.000239 19266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:44:36.003664 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "b678493d7a364555af399e7f5fbcfaca"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "6decaaa8ecec938f989fa4e659ba2cea"
server_key_iv: "5b5a9099779b3904e35a438c88820a29"
server_key_version: "encryptionkey@0"
I20260619 18:44:36.004608 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "b678493d7a364555af399e7f5fbcfaca"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "6decaaa8ecec938f989fa4e659ba2cea"
server_key_iv: "5b5a9099779b3904e35a438c88820a29"
server_key_version: "encryptionkey@0"
I20260619 18:44:36.011087 19266 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260619 18:44:36.015684 19281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:36.015817 19126 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:36.017671 19266 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:44:36.017915 19266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "b678493d7a364555af399e7f5fbcfaca"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "6decaaa8ecec938f989fa4e659ba2cea"
server_key_iv: "5b5a9099779b3904e35a438c88820a29"
server_key_version: "encryptionkey@0"
I20260619 18:44:36.018195 19266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:36.034335 19266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:36.039420 19266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:36.039750 19266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:36.040819 19266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:36.042380 19266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:36.042481 19266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 18:44:36.042614 19266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:36.042685 19266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:36.078410 19266 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:42927
I20260619 18:44:36.078512 19393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:42927 every 8 connection(s)
I20260619 18:44:36.079871 19266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:36.089674 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19266
I20260619 18:44:36.089879 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:44:36.090487 3414 external_mini_cluster.cc:1428] Setting key 47c68082c6c6b9a5b2b58ecc739006c0
Jun 19 18:44:36 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894675, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:36.103081 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.082686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:33109 (local address 127.3.85.190:32829)
0619 18:44:36.083195 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.083199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.084139 (+ 940us) server_negotiation.cc:407] Connection header received
0619 18:44:36.086069 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.086075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.086196 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.086322 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:36.088037 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.088806 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.091224 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.091894 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.092031 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.096368 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:36.096374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:36.096377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:36.096419 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:36.099498 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.100590 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.100595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.100598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.100759 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.101258 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.101261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.101263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.101548 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:36.101668 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.102577 (+ 909us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.102790 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":274,"threads_started":1}
I20260619 18:44:36.104269 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.083172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:33109)
0619 18:44:36.083946 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.083991 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.085754 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.086662 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.086671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.087231 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:36.087844 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.087862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.088976 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.088981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.091019 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.091039 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.092141 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.092151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.092416 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.093241 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:36.093265 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:36.096057 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:36.099748 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:36.099758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:36.099777 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:36.100429 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:36.100878 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:36.100882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:36.100885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:36.101129 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:36.101699 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:36.101706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:36.102119 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.102395 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.102963 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":300,"threads_started":1}
I20260619 18:44:36.106082 19394 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:36.106493 19394 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:36.107336 19394 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
I20260619 18:44:36.108978 18937 ts_manager.cc:194] Registered new tserver with Master: b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:36.109798 18937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:33109
I20260619 18:44:36.119177 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:36.137430 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.124133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.3.85.190:32829)
0619 18:44:36.124334 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.124339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.124882 (+ 543us) server_negotiation.cc:407] Connection header received
0619 18:44:36.125027 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.125031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.125100 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.125195 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:36.126713 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.127472 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.129014 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.129449 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.129559 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.132143 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:36.132149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:36.132152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:36.132191 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:36.134970 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.135684 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.135689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.135691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.135815 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.136129 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.136132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.136134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.136456 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:36.136653 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.137009 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.137189 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:44:36.141896 18937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:36.149091 18937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:36.175796 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.162600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:42503 (local address 127.0.0.1:44854)
0619 18:44:36.164255 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.164295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.164463 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.167386 (+ 2923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.167394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.167429 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:36.167827 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.167840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.170084 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.170089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.172481 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.172503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.174625 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.174632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.174894 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.174966 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.175094 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.175324 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"mutex_wait_us":50,"thread_start_us":1242,"threads_started":1}
I20260619 18:44:36.176571 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.163799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.3.85.130:33733)
0619 18:44:36.167470 (+ 3671us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.167481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.167512 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:44:36.167612 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.167619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.167906 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.168101 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:36.168853 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.170240 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.173779 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.175222 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.175360 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.175577 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.176063 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.176245 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3484,"thread_start_us":3378,"threads_started":1}
I20260619 18:44:36.176608 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.163187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:33733 (local address 127.0.0.1:49064)
0619 18:44:36.165317 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.165342 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.165488 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.168155 (+ 2667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.168163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.168207 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:36.168643 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.168656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.170410 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.170414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.173563 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.173579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.175713 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.175718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.175859 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.175882 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.175999 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.176153 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"mutex_wait_us":40,"thread_start_us":2259,"threads_started":1}
I20260619 18:44:36.177245 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.165036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.3.85.131:42927)
0619 18:44:36.168799 (+ 3763us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.168808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.168839 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:44:36.168931 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.168940 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.169164 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.169362 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:36.170365 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.171618 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.173933 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.175403 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.175534 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.175767 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.176712 (+ 945us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.176899 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3566,"thread_start_us":3401,"threads_started":1}
I20260619 18:44:36.177670 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.164710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42927 (local address 127.0.0.1:60334)
0619 18:44:36.165754 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.165771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.165883 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.169385 (+ 3502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.169390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.169417 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:36.169811 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.169823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.171780 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.171786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.173753 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.173767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.175924 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.175932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.176117 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.176154 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.176256 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.176398 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":20,"thread_start_us":304,"threads_started":1}
I20260619 18:44:36.177749 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.162762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.3.85.129:42503)
0619 18:44:36.166600 (+ 3838us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.166609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.166640 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:44:36.166765 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.166776 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.167075 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.168062 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:36.168338 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.169932 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.172684 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.174342 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.176870 (+ 2528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.177122 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.177213 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.177415 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3625,"thread_start_us":3447,"threads_started":1}
I20260619 18:44:36.181326 19061 tablet_service.cc:1467] Processing CreateTablet for tablet 36525452e05244f3a98b7788f8b14147 (DEFAULT_TABLE table=test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:36.181200 19195 tablet_service.cc:1467] Processing CreateTablet for tablet 36525452e05244f3a98b7788f8b14147 (DEFAULT_TABLE table=test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:36.181344 19329 tablet_service.cc:1467] Processing CreateTablet for tablet 36525452e05244f3a98b7788f8b14147 (DEFAULT_TABLE table=test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:36.182742 19061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36525452e05244f3a98b7788f8b14147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:36.182757 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36525452e05244f3a98b7788f8b14147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:36.182755 19329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36525452e05244f3a98b7788f8b14147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:36.195750 19413 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Bootstrap starting.
I20260619 18:44:36.198339 19414 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Bootstrap starting.
I20260619 18:44:36.199864 19413 tablet_bootstrap.cc:654] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:36.200438 19415 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Bootstrap starting.
I20260619 18:44:36.201421 19413 log.cc:824] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:36.202795 19414 tablet_bootstrap.cc:654] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:36.204288 19415 tablet_bootstrap.cc:654] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:36.204242 19414 log.cc:824] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:36.205410 19413 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: No bootstrap required, opened a new log
I20260619 18:44:36.205607 19415 log.cc:824] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:36.205756 19413 ts_tablet_manager.cc:1397] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:44:36.207830 19414 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: No bootstrap required, opened a new log
I20260619 18:44:36.208130 19414 ts_tablet_manager.cc:1397] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 18:44:36.208479 19415 tablet_bootstrap.cc:492] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: No bootstrap required, opened a new log
I20260619 18:44:36.208743 19415 ts_tablet_manager.cc:1397] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 18:44:36.217860 19415 raft_consensus.cc:348] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.217870 19413 raft_consensus.cc:348] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.218415 19415 raft_consensus.cc:374] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:36.218411 19413 raft_consensus.cc:374] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:36.218564 19415 raft_consensus.cc:729] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b678493d7a364555af399e7f5fbcfaca, State: Initialized, Role: FOLLOWER
I20260619 18:44:36.218626 19413 raft_consensus.cc:729] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0f1e0a2fe1465084bc3926b4049421, State: Initialized, Role: FOLLOWER
I20260619 18:44:36.219439 19415 consensus_queue.cc:260] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.219547 19413 consensus_queue.cc:260] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.220202 19414 raft_consensus.cc:348] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.220783 19394 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:36.220925 19414 raft_consensus.cc:374] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:36.221045 19414 raft_consensus.cc:729] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a949bde1714d96b2a156b689557992, State: Initialized, Role: FOLLOWER
I20260619 18:44:36.221628 19415 ts_tablet_manager.cc:1428] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260619 18:44:36.221976 19414 consensus_queue.cc:260] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.224174 19413 ts_tablet_manager.cc:1428] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260619 18:44:36.225402 19260 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:36.226195 19414 ts_tablet_manager.cc:1428] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Time spent starting tablet: real 0.018s user 0.017s sys 0.001s
W20260619 18:44:36.231930 19127 tablet.cc:2369] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:36.282240 19261 tablet.cc:2369] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:36.331655 19395 tablet.cc:2369] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:36.472548 19421 raft_consensus.cc:482] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:36.472892 19421 raft_consensus.cc:504] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.474788 19421 leader_election.cc:288] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503), b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:36.483021 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:39747 (local address 127.3.85.131:42927)
0619 18:44:36.476638 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.476644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.476847 (+ 203us) server_negotiation.cc:407] Connection header received
0619 18:44:36.477106 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.477114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.477191 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.477322 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:36.477770 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.478904 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.481028 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.481852 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.482072 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.482163 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.482600 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.482744 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:44:36.483089 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.476071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42927 (local address 127.3.85.130:39747)
0619 18:44:36.476716 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.476741 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.476950 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.477344 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.477347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.477367 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:36.477618 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.477624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.479063 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.479070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.480823 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.480839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.482141 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.482149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.482365 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.482394 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.482568 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.482729 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":309,"threads_started":1}
I20260619 18:44:36.483466 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.475363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:35781 (local address 127.3.85.129:42503)
0619 18:44:36.475581 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.475587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.476049 (+ 462us) server_negotiation.cc:407] Connection header received
0619 18:44:36.476352 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.476363 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.476457 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.476524 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:36.477302 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.478541 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.480991 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.481823 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.482051 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.482158 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.482993 (+ 835us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.483169 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:44:36.484174 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.475217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:42503 (local address 127.3.85.130:35781)
0619 18:44:36.475926 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:36.475949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:36.476100 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:36.476728 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:36.476731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:36.476756 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:36.477138 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.477147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.478713 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.478717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.480745 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:36.480764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.482109 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.482120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:36.482697 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.482785 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:44:36.483697 (+ 912us) client_negotiation.cc:240] Negotiation successful
0619 18:44:36.483849 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":68,"spinlock_wait_cycles":57600,"thread_start_us":400,"threads_started":1}
I20260619 18:44:36.484814 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36525452e05244f3a98b7788f8b14147" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b678493d7a364555af399e7f5fbcfaca" is_pre_election: true
I20260619 18:44:36.485409 19349 raft_consensus.cc:2454] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 0.
I20260619 18:44:36.485474 19081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36525452e05244f3a98b7788f8b14147" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0f1e0a2fe1465084bc3926b4049421" is_pre_election: true
I20260619 18:44:36.486177 19081 raft_consensus.cc:2454] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 0.
I20260619 18:44:36.486243 19151 leader_election.cc:302] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a949bde1714d96b2a156b689557992, b678493d7a364555af399e7f5fbcfaca; no voters:
I20260619 18:44:36.486871 19421 raft_consensus.cc:2781] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:36.487053 19421 raft_consensus.cc:482] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:36.487334 19421 raft_consensus.cc:3037] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:36.489811 19421 raft_consensus.cc:504] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.490540 19421 leader_election.cc:288] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 election: Requested vote from peers 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503), b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:36.491225 19081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36525452e05244f3a98b7788f8b14147" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0f1e0a2fe1465084bc3926b4049421"
I20260619 18:44:36.491325 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36525452e05244f3a98b7788f8b14147" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b678493d7a364555af399e7f5fbcfaca"
I20260619 18:44:36.491506 19081 raft_consensus.cc:3037] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:36.491573 19349 raft_consensus.cc:3037] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:36.493924 19081 raft_consensus.cc:2454] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 1.
I20260619 18:44:36.493924 19349 raft_consensus.cc:2454] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 1.
I20260619 18:44:36.494546 19151 leader_election.cc:302] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a949bde1714d96b2a156b689557992, b678493d7a364555af399e7f5fbcfaca; no voters:
I20260619 18:44:36.494923 19421 raft_consensus.cc:2781] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:36.495437 19421 raft_consensus.cc:686] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 1 LEADER]: Becoming Leader. State: Replica: 98a949bde1714d96b2a156b689557992, State: Running, Role: LEADER
I20260619 18:44:36.495954 19421 consensus_queue.cc:237] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } }
I20260619 18:44:36.501355 18936 catalog_manager.cc:5495] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a949bde1714d96b2a156b689557992 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "98a949bde1714d96b2a156b689557992" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } health_report { overall_health: HEALTHY } } }
I20260619 18:44:36.548010 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.540897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49076 (local address 127.3.85.130:33733)
0619 18:44:36.541145 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.541152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.541195 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:44:36.541292 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.541297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.541395 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.541559 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:36.541946 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.543177 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.545435 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.546074 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.546230 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.546819 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:36.547362 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:36.547492 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.547610 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.547684 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:44:36.557142 19081 raft_consensus.cc:1261] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Refusing update from remote peer 98a949bde1714d96b2a156b689557992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:36.557154 19349 raft_consensus.cc:1261] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Refusing update from remote peer 98a949bde1714d96b2a156b689557992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:36.558290 19424 consensus_queue.cc:1035] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:36.558795 19421 consensus_queue.cc:1035] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:36.572623 19428 mvcc.cc:205] Tried to move back new op lower bound from 7298640595165831168 to 7298640594932281344. Current Snapshot: MvccSnapshot[applied={T|T < 7298640595165831168}]
I20260619 18:44:36.588553 18937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:36.588904 18937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:36.592905 18937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Sending DeleteTablet for 3 replicas of tablet 36525452e05244f3a98b7788f8b14147
I20260619 18:44:36.594398 19061 tablet_service.cc:1514] Processing DeleteTablet for tablet 36525452e05244f3a98b7788f8b14147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:36 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:44854
I20260619 18:44:36.594403 19195 tablet_service.cc:1514] Processing DeleteTablet for tablet 36525452e05244f3a98b7788f8b14147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:36 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:49064
I20260619 18:44:36.594969 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18998
I20260619 18:44:36.595244 19329 tablet_service.cc:1514] Processing DeleteTablet for tablet 36525452e05244f3a98b7788f8b14147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:36 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:60334
I20260619 18:44:36.595913 19436 tablet_replica.cc:330] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: stopping tablet replica
I20260619 18:44:36.596398 19436 raft_consensus.cc:2229] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:36.596952 19436 raft_consensus.cc:2258] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:36.597797 19437 tablet_replica.cc:330] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: stopping tablet replica
I20260619 18:44:36.598245 19437 raft_consensus.cc:2229] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:36.599050 19437 raft_consensus.cc:2258] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:36.602416 19436 ts_tablet_manager.cc:1905] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:36.604485 19437 ts_tablet_manager.cc:1905] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:36.610136 19436 ts_tablet_manager.cc:1918] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:36.610343 19436 log.cc:1196] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/36525452e05244f3a98b7788f8b14147
I20260619 18:44:36.610965 19438 tablet_replica.cc:330] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: stopping tablet replica
I20260619 18:44:36.610992 19436 ts_tablet_manager.cc:1939] T 36525452e05244f3a98b7788f8b14147 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting consensus metadata
I20260619 18:44:36.611449 19438 raft_consensus.cc:2229] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:36.612016 19438 raft_consensus.cc:2258] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:36.613207 18924 catalog_manager.cc:4841] TS 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503): tablet 36525452e05244f3a98b7788f8b14147 (table test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]) successfully deleted
I20260619 18:44:36.614638 19438 ts_tablet_manager.cc:1905] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:36.617265 19437 ts_tablet_manager.cc:1918] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:36.617466 19437 log.cc:1196] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/36525452e05244f3a98b7788f8b14147
I20260619 18:44:36.618130 19437 ts_tablet_manager.cc:1939] T 36525452e05244f3a98b7788f8b14147 P 98a949bde1714d96b2a156b689557992: Deleting consensus metadata
I20260619 18:44:36.620267 18922 catalog_manager.cc:4841] TS 98a949bde1714d96b2a156b689557992 (127.3.85.130:33733): tablet 36525452e05244f3a98b7788f8b14147 (table test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]) successfully deleted
I20260619 18:44:36.621484 19438 ts_tablet_manager.cc:1918] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:36.621883 19438 log.cc:1196] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/36525452e05244f3a98b7788f8b14147
I20260619 18:44:36.622584 19438 ts_tablet_manager.cc:1939] T 36525452e05244f3a98b7788f8b14147 P b678493d7a364555af399e7f5fbcfaca: Deleting consensus metadata
I20260619 18:44:36.624383 18924 catalog_manager.cc:4841] TS b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927): tablet 36525452e05244f3a98b7788f8b14147 (table test-table [id=efb4e87cd4904d70a740a2aaf1904ae8]) successfully deleted
Jun 19 18:44:36 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:44:36.636334 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.596213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.3.85.129:42503)
0619 18:44:36.596459 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.596467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.600042 (+ 3575us) server_negotiation.cc:407] Connection header received
0619 18:44:36.600255 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.600264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.600387 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.600532 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:36.610701 (+ 10169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.611938 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.615424 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.616097 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.616209 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.629039 (+ 12830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:36.629056 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:36.629072 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:36.629138 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:36.632697 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.633882 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.633892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.633895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.634084 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.634655 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.634661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.634663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.635112 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:36.635372 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.635783 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.635977 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:44:36.637104 19121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:36.734053 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18998
I20260619 18:44:36.758522 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19132
Jun 19 18:44:36 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:44:36.776189 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.759362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.3.85.130:33733)
0619 18:44:36.759638 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.759645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.759671 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:44:36.759746 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.759756 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.759825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.759937 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:36.761270 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.762174 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.763874 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.764323 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.764477 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.769365 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:36.769383 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:36.769399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:36.769444 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:36.772746 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.773763 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.773771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.773773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.773954 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.774390 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.774393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.774395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.774869 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:36.775169 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.775550 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.775764 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 18:44:36.777118 19255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:36.894521 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19132
I20260619 18:44:36.914232 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19266
Jun 19 18:44:36 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:44:36.931156 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:36.915125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.3.85.131:42927)
0619 18:44:36.915309 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:36.915313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:36.915332 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:44:36.915416 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:36.915420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:36.915485 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:36.915568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:36.916917 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.917783 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.919420 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:36.919813 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:36.919916 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:36.925329 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:36.925348 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:36.925365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:36.925411 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:36.928408 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.929222 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.929231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.929235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.929391 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:36.929747 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:36.929750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:36.929752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:36.930124 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:36.930369 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:36.930681 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:44:36.930892 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:44:36.931876 19389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:37.022708 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19266
I20260619 18:44:37.039667 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 18905
I20260619 18:44:37.040937 18967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:37.153254 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 18905
I20260619 18:44:57.176374 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--webserver_interface=127.3.85.190
--webserver_port=37419
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:44:57.536499 19444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:57.536902 19444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:57.537081 19444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:57.545644 19444 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:44:57.545861 19444 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:44:57.545972 19444 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:44:57.546065 19444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:44:57.546166 19444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:57.546298 19444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:57.546391 19444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:44:57.546494 19444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:44:57.557386 19444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=37419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:57.558938 19444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:57.560588 19444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:57.585371 19450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:57.585348 19453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:57.585304 19451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:57.591708 19444 server_base.cc:1034] running on GCE node
I20260619 18:44:57.592815 19444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:57.594799 19444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:57.596114 19444 hybrid_clock.cc:648] HybridClock initialized: now 1781894697596038 us; error 44 us; skew 500 ppm
I20260619 18:44:57.596643 19444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:57 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894697, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:57.606326 19444 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:44:57.608659 19444 webserver.cc:460] Webserver started at http://127.3.85.190:37419/ using document root <none> and password file <none>
I20260619 18:44:57.609632 19444 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:57.609795 19444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:57.617069 19444 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:44:57.620890 19460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:57.622853 19444 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:44:57.623214 19444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "a2a7ba4faaa9417f92aca3c24be5294a"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "f761b00ff307534b89e826f66d168274"
server_key_iv: "720d5399cc86351e8f6716accb4515ac"
server_key_version: "encryptionkey@0"
I20260619 18:44:57.624830 19444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:57.645387 19444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:57.650512 19444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:57.650928 19444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:57.671672 19444 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:32829
I20260619 18:44:57.671713 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:32829 every 8 connection(s)
I20260619 18:44:57.673457 19444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:44:57.681164 19512 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:44:57.683245 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19444
I20260619 18:44:57.684231 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:42503
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=46507
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:44:57.685938 19512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap starting.
I20260619 18:44:57.707512 19512 log.cc:824] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:57.721940 19512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:44:57.722608 19512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap complete.
I20260619 18:44:57.733651 19512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:57.734715 19512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Initialized, Role: FOLLOWER
I20260619 18:44:57.735425 19512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:57.735706 19512 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:44:57.735831 19512 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:44:57.736057 19512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:44:57.740273 19512 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:57.740957 19512 leader_election.cc:302] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a7ba4faaa9417f92aca3c24be5294a; no voters:
I20260619 18:44:57.741645 19512 leader_election.cc:288] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:44:57.741847 19516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:44:57.743283 19516 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 LEADER]: Becoming Leader. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Running, Role: LEADER
I20260619 18:44:57.744029 19516 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:44:57.744397 19512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:44:57.746870 19518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a7ba4faaa9417f92aca3c24be5294a. Latest consensus state: current_term: 2 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:44:57.747385 19518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:57.747900 19517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:44:57.748282 19517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:44:57.760043 19522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:44:57.769060 19522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:44:57.770711 19522 catalog_manager.cc:1259] Loaded cluster ID: b2774c131d89449d9a20382c8aea7254
I20260619 18:44:57.770896 19522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:44:57.775378 19522 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:44:57.777177 19522 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Loaded TSK: 0
I20260619 18:44:57.825435 19522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Generated new TSK 1
I20260619 18:44:57.825943 19522 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:44:58.071758 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:58.072072 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:58.072227 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:58.080169 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:58.080302 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:58.080483 19514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:44:58.090960 19514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:42503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=46507
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:58.092389 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:58.093850 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:58.113536 19540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:58.113699 19542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:58.114130 19539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:58.115067 19514 server_base.cc:1034] running on GCE node
I20260619 18:44:58.115792 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:58.116995 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:58.118242 19514 hybrid_clock.cc:648] HybridClock initialized: now 1781894698118193 us; error 47 us; skew 500 ppm
I20260619 18:44:58.118664 19514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:58 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:58.122910 19514 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:44:58.124833 19514 webserver.cc:460] Webserver started at http://127.3.85.129:46507/ using document root <none> and password file <none>
I20260619 18:44:58.125609 19514 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:58.125777 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:58.132143 19514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:44:58.135309 19549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.136874 19514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:44:58.137084 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "9d0f1e0a2fe1465084bc3926b4049421"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "d164c6e08e1f5e93a2fc2e56482794cb"
server_key_iv: "7d9a16d84633a5f960b9499e012dc40b"
server_key_version: "encryptionkey@0"
I20260619 18:44:58.138223 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:58.155939 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:58.161018 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:58.161334 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:58.162509 19514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:58.164007 19514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:58.164098 19514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.164265 19514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:58.164350 19514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.200524 19514 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:42503
I20260619 18:44:58.200579 19661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:42503 every 8 connection(s)
I20260619 18:44:58.202193 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:44:58.211687 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19514
I20260619 18:44:58.212363 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:33733
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=37169
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:58 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:58.229044 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:58.205253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:44031 (local address 127.3.85.190:32829)
0619 18:44:58.205880 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:58.205893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:58.206676 (+ 783us) server_negotiation.cc:407] Connection header received
0619 18:44:58.209000 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:58.209015 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:58.209448 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:58.209894 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:58.211721 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.213209 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.215081 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.215794 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.216037 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:58.220103 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:58.220117 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:58.220142 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:58.220237 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:58.223883 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:58.224811 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:58.224819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:58.224825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:58.225004 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:58.225519 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:58.225523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:58.225525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:58.226059 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:58.226281 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:58.227231 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 18:44:58.227668 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":286,"threads_started":1}
I20260619 18:44:58.229043 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:58.205486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:44031)
0619 18:44:58.206474 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:58.206531 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:58.208398 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:58.210052 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:58.210066 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:58.210855 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:58.211443 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:58.211462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.213450 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.213454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:58.214885 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:58.214903 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.216093 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.216101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:58.216397 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:58.217180 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:58.217204 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:58.219718 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:58.224103 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:58.224114 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:58.224135 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:58.224619 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:58.225140 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:58.225145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:58.225149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:58.225374 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:58.226223 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:58.226232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:58.226691 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 18:44:58.226980 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 18:44:58.227625 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":419,"threads_started":1}
I20260619 18:44:58.232703 19662 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:58.233119 19662 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:58.234261 19662 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
I20260619 18:44:58.237564 19477 ts_manager.cc:194] Registered new tserver with Master: 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503)
I20260619 18:44:58.239900 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:44031
W20260619 18:44:58.563553 19666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:58.563856 19666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:58.563952 19666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:58.571453 19666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:58.571584 19666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:58.571707 19666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:44:58.582027 19666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:33733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=37169
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:58.583407 19666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:58.584746 19666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:58.598593 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:58.598697 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:58.598626 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:58.599298 19666 server_base.cc:1034] running on GCE node
I20260619 18:44:58.600003 19666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:58.601068 19666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:58.602339 19666 hybrid_clock.cc:648] HybridClock initialized: now 1781894698602275 us; error 55 us; skew 500 ppm
I20260619 18:44:58.602749 19666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:58 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:58.606135 19666 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:44:58.607980 19666 webserver.cc:460] Webserver started at http://127.3.85.130:37169/ using document root <none> and password file <none>
I20260619 18:44:58.608755 19666 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:58.608932 19666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:58.615020 19666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:44:58.618358 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.619939 19666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:44:58.620162 19666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "98a949bde1714d96b2a156b689557992"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "1cdad702017f2c294ef99440b2e3bc1e"
server_key_iv: "07d08d16541aec0c9a67f965e89decf5"
server_key_version: "encryptionkey@0"
I20260619 18:44:58.621316 19666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:58.648059 19666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:58.652801 19666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:58.653131 19666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:58.654309 19666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:58.656014 19666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:58.656153 19666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.656291 19666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:58.656378 19666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:58.693877 19666 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:33733
I20260619 18:44:58.693943 19793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:33733 every 8 connection(s)
I20260619 18:44:58.695436 19666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:44:58.698900 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19666
I20260619 18:44:58.699769 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:42927
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=36701
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:44:58 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:58.724678 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:58.698260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:40057 (local address 127.3.85.190:32829)
0619 18:44:58.698531 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:58.698538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:58.700030 (+ 1492us) server_negotiation.cc:407] Connection header received
0619 18:44:58.703337 (+ 3307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:58.703343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:58.703444 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:58.703542 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:58.706003 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.707310 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.710861 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.711561 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.711681 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:58.716860 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:58.716869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:58.716877 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:58.716933 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:58.720726 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:58.721738 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:58.721747 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:58.721754 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:58.721932 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:58.722485 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:58.722493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:58.722499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:58.722917 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:58.723088 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:58.723878 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 18:44:58.724203 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:44:58.725520 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:58.698814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:40057)
0619 18:44:58.699816 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:58.699884 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:58.702901 (+ 3017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:58.703879 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:58.703894 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:58.704770 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:58.705687 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:58.705718 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.707565 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.707573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:58.710610 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:58.710639 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:58.712111 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:58.712119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:58.712428 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:58.713416 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:58.713449 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:58.716532 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:58.720991 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:58.721000 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:58.721016 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:58.721558 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:58.722088 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:58.722092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:58.722094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:58.722319 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:58.723053 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:58.723060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:58.723447 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 18:44:58.723697 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 18:44:58.724142 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":406,"threads_started":1}
I20260619 18:44:58.727365 19794 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:58.727730 19794 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:58.728494 19794 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
I20260619 18:44:58.730355 19477 ts_manager.cc:194] Registered new tserver with Master: 98a949bde1714d96b2a156b689557992 (127.3.85.130:33733)
I20260619 18:44:58.731323 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:40057
W20260619 18:44:59.064405 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:44:59.064759 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:44:59.064936 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:44:59.072842 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:44:59.073046 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:44:59.073259 19797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:44:59.083487 19797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=36701
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:44:59.084913 19797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:44:59.086259 19797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:44:59.100272 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.100253 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:44:59.100369 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:44:59.101223 19797 server_base.cc:1034] running on GCE node
I20260619 18:44:59.101892 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:44:59.103034 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:44:59.104249 19797 hybrid_clock.cc:648] HybridClock initialized: now 1781894699104184 us; error 60 us; skew 500 ppm
I20260619 18:44:59.104579 19797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:44:59 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:44:59.107980 19797 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:44:59.109707 19797 webserver.cc:460] Webserver started at http://127.3.85.131:36701/ using document root <none> and password file <none>
I20260619 18:44:59.110558 19797 fs_manager.cc:359] Metadata directory not provided
I20260619 18:44:59.110733 19797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:44:59.116690 19797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:44:59.119915 19812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.121371 19797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:44:59.121578 19797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "b678493d7a364555af399e7f5fbcfaca"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "6decaaa8ecec938f989fa4e659ba2cea"
server_key_iv: "5b5a9099779b3904e35a438c88820a29"
server_key_version: "encryptionkey@0"
I20260619 18:44:59.122756 19797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:44:59.140553 19797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:44:59.145452 19797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:44:59.145788 19797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:44:59.146907 19797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:44:59.148445 19797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:44:59.148591 19797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.148723 19797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:44:59.148814 19797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:44:59.184742 19797 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:42927
I20260619 18:44:59.184837 19924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:42927 every 8 connection(s)
I20260619 18:44:59.186308 19797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:44:59.194128 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19797
Jun 19 18:44:59 dist-test-slave-09fp krb5kdc[18892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894699, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:44:59.210196 19664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.189203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:57707 (local address 127.3.85.190:32829)
0619 18:44:59.189436 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.189441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.190734 (+ 1293us) server_negotiation.cc:407] Connection header received
0619 18:44:59.192741 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.192746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.192851 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.192970 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.194809 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.195801 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.197936 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.198759 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.199020 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.203380 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.203385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.203389 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.203418 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.206300 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.207422 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.207426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.207429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.207579 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.208133 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.208143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.208147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.208474 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.208714 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.209462 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.209804 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":105}
I20260619 18:44:59.211315 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.189677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:57707)
0619 18:44:59.190547 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.190594 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.192410 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.193242 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.193251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.193835 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:44:59.194597 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.194621 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.196031 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.196035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.197723 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.197740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.199062 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.199072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.199437 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.200370 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:44:59.200395 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:44:59.203069 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:44:59.206539 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.206550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.206568 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.207248 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.207754 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:44:59.207758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:44:59.207760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:44:59.207977 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:44:59.208640 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:44:59.208649 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:44:59.209056 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.209363 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.209966 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260619 18:44:59.213181 19925 heartbeater.cc:347] Connected to a master server at 127.3.85.190:32829
I20260619 18:44:59.213342 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.195275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.3.85.190:32829)
0619 18:44:59.195880 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.195887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.196271 (+ 384us) server_negotiation.cc:407] Connection header received
0619 18:44:59.196416 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.196420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.196497 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.196610 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.198861 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.200120 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.202199 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.202926 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.203078 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.206966 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.206976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.206984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.207028 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.209930 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.210971 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.210980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.210984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.211119 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.211626 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.211631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.211634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.211966 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.212222 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.212633 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.212884 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":472,"thread_start_us":294,"threads_started":1}
I20260619 18:44:59.213593 19925 heartbeater.cc:464] Registering TS with master...
I20260619 18:44:59.214695 19925 heartbeater.cc:511] Master 127.3.85.190:32829 requested a full tablet report, sending...
I20260619 18:44:59.216877 19477 ts_manager.cc:194] Registered new tserver with Master: b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:59.217783 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:57707
I20260619 18:44:59.229751 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:44:59.236888 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.230862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.3.85.190:32829)
0619 18:44:59.231131 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.231142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.231574 (+ 432us) server_negotiation.cc:407] Connection header received
0619 18:44:59.231737 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.231741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.231899 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.232048 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:59.232453 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.233202 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.234844 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.235302 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.235400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.235993 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:59.236222 (+ 229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 18:44:59.236580 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54822: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:44:59.237066 19928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54822: token expired
W20260619 18:44:59.237752 19399 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.85.190:32829)
I20260619 18:44:59.244294 19662 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:59.251921 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.238368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.3.85.190:32829)
0619 18:44:59.238602 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.238606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.238625 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:44:59.238670 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.238674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.238734 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.238816 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.240256 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.241008 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.242526 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.243042 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.243207 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.246505 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.246516 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.246520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.246553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.249043 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.249768 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.249775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.249779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.249923 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.250321 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.250326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.250331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.250742 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.250994 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.251333 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.251569 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:44:59.255534 19477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:44:59.263581 19477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:44:59.291482 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.277210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:33733 (local address 127.0.0.1:56196)
0619 18:44:59.280788 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.280846 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.281164 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.282477 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.282491 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.282541 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:59.283169 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.283190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.285326 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.285332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.287971 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.287990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.290591 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.290609 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.290810 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.290854 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.290980 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.291144 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3401,"mutex_wait_us":1,"thread_start_us":2369,"threads_started":1}
I20260619 18:44:59.291642 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.277818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.3.85.130:33733)
0619 18:44:59.278403 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.278418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.280986 (+ 2568us) server_negotiation.cc:407] Connection header received
0619 18:44:59.281491 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.281502 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.281823 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.282016 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:59.283481 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.285103 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.288218 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.289828 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.289968 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.290201 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.291102 (+ 901us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.291307 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":299,"threads_started":1}
I20260619 18:44:59.294497 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.277087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:42503 (local address 127.0.0.1:57550)
0619 18:44:59.281531 (+ 4444us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.281563 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.281684 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.286745 (+ 5061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.286752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.286799 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:59.287234 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.287246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.289138 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.289148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.291466 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.291485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.293539 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.293547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.293730 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.293767 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.293863 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.294020 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2637,"mutex_wait_us":24,"thread_start_us":3344,"threads_started":1}
I20260619 18:44:59.295117 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.281523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.3.85.131:42927)
0619 18:44:59.283986 (+ 2463us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.284979 (+ 993us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.285041 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:44:59.285171 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.285183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.285555 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.285781 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:59.286588 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.287971 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.291915 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.293276 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.293467 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.293759 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.294630 (+ 871us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.294833 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2251,"thread_start_us":2054,"threads_started":1}
I20260619 18:44:59.295352 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.277238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.3.85.129:42503)
0619 18:44:59.286078 (+ 8840us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.286091 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.286125 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:44:59.286230 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.286240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.286511 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.286738 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:59.287434 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.288926 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.291976 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.293153 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.294339 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.294771 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.294849 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.295020 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8635,"thread_start_us":8478,"threads_started":1}
I20260619 18:44:59.295607 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.281966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42927 (local address 127.0.0.1:46712)
0619 18:44:59.284398 (+ 2432us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.284426 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.284605 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.286031 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.286036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.286066 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:59.286372 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.286380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.288118 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.288126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.291690 (+ 3564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.291717 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.293923 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.293929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.294076 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.294106 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.294189 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.294314 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2217,"mutex_wait_us":3,"thread_start_us":2777,"threads_started":1}
I20260619 18:44:59.296283 19729 tablet_service.cc:1467] Processing CreateTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (DEFAULT_TABLE table=test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:59.298532 19729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b8549539cf4cbc8d3efa014b6b3fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.299093 19860 tablet_service.cc:1467] Processing CreateTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (DEFAULT_TABLE table=test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:59.299243 19597 tablet_service.cc:1467] Processing CreateTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (DEFAULT_TABLE table=test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:44:59.300328 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b8549539cf4cbc8d3efa014b6b3fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.300591 19597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b8549539cf4cbc8d3efa014b6b3fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:44:59.313776 19941 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Bootstrap starting.
I20260619 18:44:59.316027 19942 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Bootstrap starting.
I20260619 18:44:59.317884 19943 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Bootstrap starting.
I20260619 18:44:59.318029 19941 tablet_bootstrap.cc:654] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.319947 19942 tablet_bootstrap.cc:654] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.320302 19941 log.cc:824] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.321308 19942 log.cc:824] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.321607 19943 tablet_bootstrap.cc:654] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Neither blocks nor log segments found. Creating new log.
I20260619 18:44:59.322847 19943 log.cc:824] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Log is configured to *not* fsync() on all Append() calls
I20260619 18:44:59.324033 19941 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: No bootstrap required, opened a new log
I20260619 18:44:59.324415 19941 ts_tablet_manager.cc:1397] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:44:59.325119 19942 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: No bootstrap required, opened a new log
I20260619 18:44:59.325392 19942 ts_tablet_manager.cc:1397] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260619 18:44:59.325721 19943 tablet_bootstrap.cc:492] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: No bootstrap required, opened a new log
I20260619 18:44:59.326084 19943 ts_tablet_manager.cc:1397] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 18:44:59.334265 19943 raft_consensus.cc:348] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.334801 19943 raft_consensus.cc:374] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.334916 19943 raft_consensus.cc:729] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0f1e0a2fe1465084bc3926b4049421, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.335623 19943 consensus_queue.cc:260] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.336408 19941 raft_consensus.cc:348] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.336409 19942 raft_consensus.cc:348] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.336874 19942 raft_consensus.cc:374] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.336943 19941 raft_consensus.cc:374] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:44:59.337021 19942 raft_consensus.cc:729] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a949bde1714d96b2a156b689557992, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.337086 19941 raft_consensus.cc:729] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b678493d7a364555af399e7f5fbcfaca, State: Initialized, Role: FOLLOWER
I20260619 18:44:59.337555 19943 ts_tablet_manager.cc:1428] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 18:44:59.337837 19941 consensus_queue.cc:260] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.337841 19942 consensus_queue.cc:260] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.339169 19925 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:59.339593 19794 heartbeater.cc:503] Master 127.3.85.190:32829 was elected leader, sending a full tablet report...
I20260619 18:44:59.340307 19942 ts_tablet_manager.cc:1428] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260619 18:44:59.341807 19941 ts_tablet_manager.cc:1428] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260619 18:44:59.403124 19948 raft_consensus.cc:482] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:44:59.403450 19948 raft_consensus.cc:504] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.405547 19948 leader_election.cc:288] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503), b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:59.413044 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.405992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:41235 (local address 127.3.85.129:42503)
0619 18:44:59.406208 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.406213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.406934 (+ 721us) server_negotiation.cc:407] Connection header received
0619 18:44:59.407287 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.407294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.407377 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.407450 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:59.408105 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.408853 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.411089 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.411940 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.412175 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.412268 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.412619 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.412769 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:44:59.413081 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.407096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:34327 (local address 127.3.85.131:42927)
0619 18:44:59.407312 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.407318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.407628 (+ 310us) server_negotiation.cc:407] Connection header received
0619 18:44:59.407857 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.407862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.407933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.408034 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:44:59.408567 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.409302 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.411227 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.412055 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.412269 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.412348 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.412660 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.412809 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:44:59.413232 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.405952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:42503 (local address 127.3.85.130:41235)
0619 18:44:59.406831 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.406851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.407065 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.407614 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.407618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.407637 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:59.407947 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.407955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.409032 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.409044 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.410849 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.410866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.412221 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.412232 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.412432 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.412458 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.412586 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.412752 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":1,"thread_start_us":511,"threads_started":1}
I20260619 18:44:59.413527 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.406918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:42927 (local address 127.3.85.130:34327)
0619 18:44:59.407515 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 18:44:59.407529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:44:59.407708 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:44:59.408048 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:44:59.408053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:44:59.408075 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:44:59.408419 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.408429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.409456 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.409463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.411036 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:44:59.411050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.412306 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.412312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:44:59.412483 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.412506 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:44:59.412617 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:44:59.412757 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":285,"threads_started":1}
I20260619 18:44:59.414533 19617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b8549539cf4cbc8d3efa014b6b3fa8" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0f1e0a2fe1465084bc3926b4049421" is_pre_election: true
I20260619 18:44:59.414675 19880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b8549539cf4cbc8d3efa014b6b3fa8" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b678493d7a364555af399e7f5fbcfaca" is_pre_election: true
I20260619 18:44:59.415133 19617 raft_consensus.cc:2454] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 0.
I20260619 18:44:59.415190 19880 raft_consensus.cc:2454] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 0.
I20260619 18:44:59.415946 19685 leader_election.cc:302] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a949bde1714d96b2a156b689557992, 9d0f1e0a2fe1465084bc3926b4049421; no voters:
I20260619 18:44:59.416373 19948 raft_consensus.cc:2781] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:44:59.416535 19948 raft_consensus.cc:482] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:44:59.416651 19948 raft_consensus.cc:3037] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.418998 19948 raft_consensus.cc:504] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.419670 19948 leader_election.cc:288] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 election: Requested vote from peers 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503), b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927)
I20260619 18:44:59.420266 19617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b8549539cf4cbc8d3efa014b6b3fa8" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0f1e0a2fe1465084bc3926b4049421"
I20260619 18:44:59.420320 19880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78b8549539cf4cbc8d3efa014b6b3fa8" candidate_uuid: "98a949bde1714d96b2a156b689557992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b678493d7a364555af399e7f5fbcfaca"
I20260619 18:44:59.420540 19617 raft_consensus.cc:3037] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.420563 19880 raft_consensus.cc:3037] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:44:59.423169 19617 raft_consensus.cc:2454] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 1.
I20260619 18:44:59.423169 19880 raft_consensus.cc:2454] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a949bde1714d96b2a156b689557992 in term 1.
I20260619 18:44:59.423729 19685 leader_election.cc:302] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a949bde1714d96b2a156b689557992, b678493d7a364555af399e7f5fbcfaca; no voters:
I20260619 18:44:59.424115 19948 raft_consensus.cc:2781] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:44:59.424618 19948 raft_consensus.cc:686] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 1 LEADER]: Becoming Leader. State: Replica: 98a949bde1714d96b2a156b689557992, State: Running, Role: LEADER
I20260619 18:44:59.425426 19948 consensus_queue.cc:237] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } }
I20260619 18:44:59.430501 19477 catalog_manager.cc:5495] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a949bde1714d96b2a156b689557992 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "98a949bde1714d96b2a156b689557992" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a949bde1714d96b2a156b689557992" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 33733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 } health_report { overall_health: UNKNOWN } } }
W20260619 18:44:59.438086 19926 tablet.cc:2369] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:59.447672 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.441691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.3.85.130:33733)
0619 18:44:59.441983 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.441989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.442012 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:44:59.442201 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.442205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.442298 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.442499 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:44:59.442990 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.443795 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.445357 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.445782 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.445888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.446417 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:44:59.447018 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:44:59.447146 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.447238 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.447314 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 18:44:59.447633 19795 tablet.cc:2369] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:44:59.454770 19663 tablet.cc:2369] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:44:59.456238 19617 raft_consensus.cc:1261] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Refusing update from remote peer 98a949bde1714d96b2a156b689557992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:59.456238 19880 raft_consensus.cc:1261] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Refusing update from remote peer 98a949bde1714d96b2a156b689557992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:44:59.457279 19952 consensus_queue.cc:1035] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0f1e0a2fe1465084bc3926b4049421" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:59.457708 19948 consensus_queue.cc:1035] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b678493d7a364555af399e7f5fbcfaca" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:44:59.473518 19956 mvcc.cc:205] Tried to move back new op lower bound from 7298640688961404928 to 7298640688851152896. Current Snapshot: MvccSnapshot[applied={T|T < 7298640688961404928}]
I20260619 18:44:59.486927 19477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:59.487241 19477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:44:59.491356 19477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Sending DeleteTablet for 3 replicas of tablet 78b8549539cf4cbc8d3efa014b6b3fa8
I20260619 18:44:59.492792 19860 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:46712
I20260619 18:44:59.492857 19729 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:56196
I20260619 18:44:59.492945 19597 tablet_service.cc:1514] Processing DeleteTablet for tablet 78b8549539cf4cbc8d3efa014b6b3fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:44:59 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:57550
I20260619 18:44:59.493700 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19514
I20260619 18:44:59.505460 19966 tablet_replica.cc:330] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: stopping tablet replica
I20260619 18:44:59.505753 19964 tablet_replica.cc:330] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: stopping tablet replica
I20260619 18:44:59.506197 19966 raft_consensus.cc:2229] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:59.506197 19964 raft_consensus.cc:2229] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:44:59.506891 19966 raft_consensus.cc:2258] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:59.506896 19964 raft_consensus.cc:2258] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:59.509279 19964 ts_tablet_manager.cc:1905] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:59.509279 19966 ts_tablet_manager.cc:1905] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:59.517256 19964 ts_tablet_manager.cc:1918] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:59.517261 19966 ts_tablet_manager.cc:1918] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:59.517530 19964 log.cc:1196] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/78b8549539cf4cbc8d3efa014b6b3fa8
I20260619 18:44:59.517570 19966 log.cc:1196] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/78b8549539cf4cbc8d3efa014b6b3fa8
I20260619 18:44:59.518224 19966 ts_tablet_manager.cc:1939] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 9d0f1e0a2fe1465084bc3926b4049421: Deleting consensus metadata
I20260619 18:44:59.518224 19964 ts_tablet_manager.cc:1939] T 78b8549539cf4cbc8d3efa014b6b3fa8 P b678493d7a364555af399e7f5fbcfaca: Deleting consensus metadata
I20260619 18:44:59.520387 19464 catalog_manager.cc:4841] TS b678493d7a364555af399e7f5fbcfaca (127.3.85.131:42927): tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (table test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]) successfully deleted
I20260619 18:44:59.520933 19464 catalog_manager.cc:4841] TS 9d0f1e0a2fe1465084bc3926b4049421 (127.3.85.129:42503): tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (table test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]) successfully deleted
I20260619 18:44:59.521569 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.495764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.3.85.129:42503)
0619 18:44:59.495966 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.495971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.495991 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:44:59.496051 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.496057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.496171 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.496275 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.498638 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.499879 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.501945 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.502693 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.502841 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.511179 (+ 8338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.511201 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.511224 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.511287 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.515822 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.517025 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.517036 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.517040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.517243 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.518795 (+ 1552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.518815 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.518819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.519356 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.519708 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.520884 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.521158 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:44:59.523159 19965 tablet_replica.cc:330] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: stopping tablet replica
I20260619 18:44:59.523329 19657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:59.523679 19965 raft_consensus.cc:2229] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:44:59.524493 19965 raft_consensus.cc:2258] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:44:59.526716 19965 ts_tablet_manager.cc:1905] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:44:59.534919 19965 ts_tablet_manager.cc:1918] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:44:59.535148 19965 log.cc:1196] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/78b8549539cf4cbc8d3efa014b6b3fa8
I20260619 18:44:59.535775 19965 ts_tablet_manager.cc:1939] T 78b8549539cf4cbc8d3efa014b6b3fa8 P 98a949bde1714d96b2a156b689557992: Deleting consensus metadata
I20260619 18:44:59.537895 19462 catalog_manager.cc:4841] TS 98a949bde1714d96b2a156b689557992 (127.3.85.130:33733): tablet 78b8549539cf4cbc8d3efa014b6b3fa8 (table test-table [id=a4c7dc31452e4417b6fb60c8baec22ac]) successfully deleted
I20260619 18:44:59.619045 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19514
I20260619 18:44:59.643998 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19666
I20260619 18:44:59.662250 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.645067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.3.85.130:33733)
0619 18:44:59.645262 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.645267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.645287 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:44:59.645383 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.645386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.645457 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.645546 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.647532 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.648386 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.650510 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.650990 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.651110 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.654907 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.654925 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.654946 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.654998 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.658646 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.659510 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.659522 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.659527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.659753 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.660298 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.660304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.660308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.660813 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.661191 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.661560 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.661813 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:44:59.663225 19789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:59.789633 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19666
I20260619 18:44:59.809221 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19797
I20260619 18:44:59.823652 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:44:59.810169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.3.85.131:42927)
0619 18:44:59.810386 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:44:59.810391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:44:59.810410 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:44:59.810596 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:44:59.810600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:44:59.810680 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:44:59.810780 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:44:59.812191 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.812957 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.814273 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:44:59.814869 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:44:59.814983 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:44:59.817763 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:44:59.817780 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:44:59.817796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:44:59.817835 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:44:59.820739 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.821500 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.821510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.821513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.821689 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:44:59.822116 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:44:59.822119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:44:59.822122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:44:59.822572 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:44:59.822811 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:44:59.823212 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 18:44:59.823405 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:44:59.824332 19920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:44:59.914775 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19797
I20260619 18:44:59.932035 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 19444
I20260619 18:44:59.933374 19507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:00.052505 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 19444
I20260619 18:45:20.075798 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--webserver_interface=127.3.85.190
--webserver_port=37419
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:20.435195 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:20.435545 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:20.435699 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:20.444169 19971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:45:20.444304 19971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:45:20.444344 19971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:45:20.444429 19971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:20.444525 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:20.444649 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:20.444692 19971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:20.444741 19971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:20.456099 19971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=37419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:20.457559 19971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:20.459090 19971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:20.473614 19977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:20.473731 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:20.473778 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:20.475733 19971 server_base.cc:1034] running on GCE node
I20260619 18:45:20.476791 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:20.478559 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:20.479859 19971 hybrid_clock.cc:648] HybridClock initialized: now 1781894720479786 us; error 44 us; skew 500 ppm
I20260619 18:45:20.480252 19971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:20 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894720, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:20.492226 19971 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:45:20.495844 19971 webserver.cc:460] Webserver started at http://127.3.85.190:37419/ using document root <none> and password file <none>
I20260619 18:45:20.497043 19971 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:20.497253 19971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:20.505509 19971 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:45:20.510339 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:20.512583 19971 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260619 18:45:20.512871 19971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "a2a7ba4faaa9417f92aca3c24be5294a"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "f761b00ff307534b89e826f66d168274"
server_key_iv: "720d5399cc86351e8f6716accb4515ac"
server_key_version: "encryptionkey@0"
I20260619 18:45:20.514420 19971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:20.543934 19971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:20.548971 19971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:20.549320 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:20.572037 19971 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:32829
I20260619 18:45:20.572074 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:32829 every 8 connection(s)
I20260619 18:45:20.574636 19971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:20.582669 20038 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:45:20.583498 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 19971
I20260619 18:45:20.584556 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:42503
--local_ip_for_outbound_sockets=127.3.85.129
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=46507
--webserver_interface=127.3.85.129
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:20.588248 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap starting.
I20260619 18:45:20.612938 20038 log.cc:824] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:20.634281 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:45:20.635048 20038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Bootstrap complete.
I20260619 18:45:20.645222 20038 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:45:20.646317 20038 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Initialized, Role: FOLLOWER
I20260619 18:45:20.647105 20038 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:45:20.647397 20038 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:20.647541 20038 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:20.647738 20038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:45:20.651919 20038 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:45:20.652488 20038 leader_election.cc:302] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a7ba4faaa9417f92aca3c24be5294a; no voters:
I20260619 18:45:20.653242 20038 leader_election.cc:288] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:45:20.653388 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:45:20.654594 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [term 3 LEADER]: Becoming Leader. State: Replica: a2a7ba4faaa9417f92aca3c24be5294a, State: Running, Role: LEADER
I20260619 18:45:20.655262 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } }
I20260619 18:45:20.655697 20038 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:20.657963 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a7ba4faaa9417f92aca3c24be5294a. Latest consensus state: current_term: 3 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:45:20.658425 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:20.658875 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a7ba4faaa9417f92aca3c24be5294a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 32829 } } }
I20260619 18:45:20.659305 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:20.667873 20051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:20.677449 20051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:20.679776 20051 catalog_manager.cc:1259] Loaded cluster ID: b2774c131d89449d9a20382c8aea7254
I20260619 18:45:20.680034 20051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:20.684304 20059 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:45:20.685427 20051 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:20.686988 20051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Loaded TSK: 0
I20260619 18:45:20.687160 20051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2a7ba4faaa9417f92aca3c24be5294a: Loaded TSK: 1
F20260619 18:45:20.688053 20051 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f55918b09ad google::LogMessage::Fail() at ??:0
@ 0x7f55918b388a google::LogMessage::SendToLog() at ??:0
@ 0x7f55918b04de google::LogMessage::Flush() at ??:0
@ 0x7f55918b1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f55956147d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f559560373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f55a72df1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f55a72de783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f55a736a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f55a736a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f55a73b45e9 std::function<>::operator()() at ??:0
@ 0x7f55a72e2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f55a72e0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f55a7368610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f55a7368439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55944fc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f55944ff91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f55944ff799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55944daeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55977b46db start_thread at ??:0
@ 0x7f559074a61f clone at ??:0
W20260619 18:45:21.019485 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:21.019815 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:21.019917 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:21.027899 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:21.028038 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:21.028159 20040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:21.038504 20040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:42503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=46507
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:21.039888 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:21.041285 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:21.055862 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:21.055790 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:21.055908 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:21.056874 20040 server_base.cc:1034] running on GCE node
I20260619 18:45:21.057569 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:21.058825 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:21.060111 20040 hybrid_clock.cc:648] HybridClock initialized: now 1781894721060033 us; error 70 us; skew 500 ppm
I20260619 18:45:21.060503 20040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:21 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:21.064077 20040 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:45:21.065801 20040 webserver.cc:460] Webserver started at http://127.3.85.129:46507/ using document root <none> and password file <none>
I20260619 18:45:21.066633 20040 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:21.066795 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:21.072700 20040 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 18:45:21.075956 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.077502 20040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:21.077728 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "9d0f1e0a2fe1465084bc3926b4049421"
format_stamp: "Formatted at 2026-06-19 18:44:34 on dist-test-slave-09fp"
server_key: "d164c6e08e1f5e93a2fc2e56482794cb"
server_key_iv: "7d9a16d84633a5f960b9499e012dc40b"
server_key_version: "encryptionkey@0"
I20260619 18:45:21.078969 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:21.106989 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:21.112314 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:21.112676 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:21.113894 20040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:21.115499 20040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:21.115643 20040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.115793 20040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:21.115892 20040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.152845 20040 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:42503
I20260619 18:45:21.152980 20207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:42503 every 8 connection(s)
I20260619 18:45:21.154893 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:21.156265 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20040
I20260619 18:45:21.156982 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:33733
--local_ip_for_outbound_sockets=127.3.85.130
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=37169
--webserver_interface=127.3.85.130
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:21.161113 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.158540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:35979)
0619 18:45:21.159880 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.160120 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":511,"threads_started":1}
W20260619 18:45:21.162026 20208 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:32829 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:32829: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
I20260619 18:45:21.163087 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.162707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:42927)
0619 18:45:21.162883 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.162967 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:45:21.164340 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.163904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:57297)
0619 18:45:21.164080 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.164182 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:45:21.164616 20208 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:45:21.506883 20211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:21.507182 20211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:21.507366 20211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:21.514992 20211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:21.515125 20211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:21.515332 20211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:21.525763 20211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:33733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=37169
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:21.527303 20211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:21.528673 20211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:21.542981 20216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:21.542958 20217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:21.543053 20219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:21.543815 20211 server_base.cc:1034] running on GCE node
I20260619 18:45:21.544523 20211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:21.545663 20211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:21.546928 20211 hybrid_clock.cc:648] HybridClock initialized: now 1781894721546876 us; error 56 us; skew 500 ppm
I20260619 18:45:21.547302 20211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:21 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:21.550825 20211 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:45:21.552588 20211 webserver.cc:460] Webserver started at http://127.3.85.130:37169/ using document root <none> and password file <none>
I20260619 18:45:21.553355 20211 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:21.553515 20211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:21.559799 20211 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:45:21.563235 20226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.564745 20211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:21.564954 20211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "98a949bde1714d96b2a156b689557992"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "1cdad702017f2c294ef99440b2e3bc1e"
server_key_iv: "07d08d16541aec0c9a67f965e89decf5"
server_key_version: "encryptionkey@0"
I20260619 18:45:21.566047 20211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:21.594851 20211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:21.599928 20211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:21.600263 20211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:21.601421 20211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:21.602960 20211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:21.603087 20211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.603226 20211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:21.603324 20211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:21.640328 20211 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:33733
I20260619 18:45:21.640435 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:33733 every 8 connection(s)
I20260619 18:45:21.641875 20211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:21.644933 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20211
I20260619 18:45:21.645586 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:42927
--local_ip_for_outbound_sockets=127.3.85.131
--tserver_master_addrs=127.3.85.190:32829
--webserver_port=36701
--webserver_interface=127.3.85.131
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:45:21.648161 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.645365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:47977)
0619 18:45:21.646340 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.646695 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":406,"threads_started":1}
W20260619 18:45:21.648985 20339 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:32829 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:32829: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
I20260619 18:45:21.650167 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.649665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:51913)
0619 18:45:21.649950 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.650034 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 18:45:21.651391 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:21.650988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:53039)
0619 18:45:21.651159 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:21.651257 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:45:21.651686 20339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:45:22.013901 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:22.014235 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:22.014335 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:22.022210 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:22.022336 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:22.022501 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:22.033186 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:42927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=36701
--tserver_master_addrs=127.3.85.190:32829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:22.034693 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:22.036065 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:22.052052 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:22.052120 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:22.052083 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:22.052939 20342 server_base.cc:1034] running on GCE node
I20260619 18:45:22.053558 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:22.054715 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:22.055969 20342 hybrid_clock.cc:648] HybridClock initialized: now 1781894722055905 us; error 51 us; skew 500 ppm
I20260619 18:45:22.056346 20342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:22 dist-test-slave-09fp krb5kdc[18892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:22.059965 20342 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:45:22.061827 20342 webserver.cc:460] Webserver started at http://127.3.85.131:36701/ using document root <none> and password file <none>
I20260619 18:45:22.062805 20342 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:22.062966 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:22.069093 20342 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:45:22.072700 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:22.074301 20342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:45:22.074594 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "b678493d7a364555af399e7f5fbcfaca"
format_stamp: "Formatted at 2026-06-19 18:44:35 on dist-test-slave-09fp"
server_key: "6decaaa8ecec938f989fa4e659ba2cea"
server_key_iv: "5b5a9099779b3904e35a438c88820a29"
server_key_version: "encryptionkey@0"
I20260619 18:45:22.075723 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:22.129256 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:22.134063 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:22.134397 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:22.135555 20342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:22.137054 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:22.137194 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:22.137353 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:22.137445 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:22.166177 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.165298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:58779)
0619 18:45:22.165821 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:22.165919 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
I20260619 18:45:22.174068 20342 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:42927
I20260619 18:45:22.174161 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:42927 every 8 connection(s)
I20260619 18:45:22.175661 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:22.180963 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.178813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:54401)
0619 18:45:22.179621 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:22.179857 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":297,"threads_started":1}
W20260619 18:45:22.181669 20471 heartbeater.cc:644] Failed to heartbeat to 127.3.85.190:32829 (0 consecutive failures): Network error: Failed to ping master at 127.3.85.190:32829: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
I20260619 18:45:22.182796 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.182322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:40729)
0619 18:45:22.182603 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:22.182676 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:45:22.183881 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.183386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:33955)
0619 18:45:22.183657 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:22.183757 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 18:45:22.184140 20471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:45:22.184383 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20342
I20260619 18:45:22.653435 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:22.652415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:47551)
0619 18:45:22.653066 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:22.653184 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":336,"threads_started":1}
I20260619 18:45:23.167871 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.167121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:56811)
0619 18:45:23.167593 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.167689 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":239,"threads_started":1}
I20260619 18:45:23.185912 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.184868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:36227)
0619 18:45:23.185541 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.185651 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":368,"threads_started":1}
I20260619 18:45:23.655611 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:23.654699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:41733)
0619 18:45:23.655290 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:23.655414 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260619 18:45:24.169926 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:24.169002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:41903)
0619 18:45:24.169557 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:24.169734 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":286,"threads_started":1}
I20260619 18:45:24.187899 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:24.187110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:39759)
0619 18:45:24.187640 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:24.187732 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
I20260619 18:45:24.657418 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:24.656662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:46139)
0619 18:45:24.657143 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:24.657237 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260619 18:45:25.171797 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:25.171045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:33477)
0619 18:45:25.171516 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:25.171603 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260619 18:45:25.189791 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:25.188979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:54563)
0619 18:45:25.189502 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:25.189590 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
I20260619 18:45:25.659554 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:25.658607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:41551)
0619 18:45:25.659180 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:25.659269 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":295,"threads_started":1}
I20260619 18:45:26.173614 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:26.172801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:59651)
0619 18:45:26.173325 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:26.173413 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":264,"threads_started":1}
I20260619 18:45:26.191843 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:26.190974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:51421)
0619 18:45:26.191536 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:26.191655 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":317,"threads_started":1}
I20260619 18:45:26.661501 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:26.660645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:39589)
0619 18:45:26.661145 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:26.661236 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260619 18:45:27.175519 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:27.174716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:60571)
0619 18:45:27.175234 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:27.175340 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260619 18:45:27.193740 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:27.192955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:50045)
0619 18:45:27.193446 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:27.193553 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":255,"threads_started":1}
I20260619 18:45:27.663602 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:27.662631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:52077)
0619 18:45:27.663266 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:27.663377 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":352,"threads_started":1}
I20260619 18:45:28.177345 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:28.176523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:53213)
0619 18:45:28.177046 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:28.177141 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260619 18:45:28.195701 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:28.194850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:52001)
0619 18:45:28.195426 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:28.195529 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":323,"threads_started":1}
I20260619 18:45:28.666055 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:28.664837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:58741)
0619 18:45:28.665556 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:28.665685 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":320,"threads_started":1}
I20260619 18:45:29.179076 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:29.178322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:42623)
0619 18:45:29.178831 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:29.178917 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":295,"threads_started":1}
I20260619 18:45:29.197578 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:29.196707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:42277)
0619 18:45:29.197210 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:29.197363 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":215,"threads_started":1}
I20260619 18:45:29.667930 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:29.667122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:44573)
0619 18:45:29.667626 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:29.667721 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":262,"threads_started":1}
I20260619 18:45:30.181016 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:30.180042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:37485)
0619 18:45:30.180696 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:30.180790 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":361,"threads_started":1}
I20260619 18:45:30.199507 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:30.198688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:34247)
0619 18:45:30.199223 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:30.199331 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":296,"threads_started":1}
I20260619 18:45:30.669647 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:30.668874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:35497)
0619 18:45:30.669374 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:30.669461 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
I20260619 18:45:31.182901 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:31.182079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:35893)
0619 18:45:31.182635 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:31.182726 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":274,"threads_started":1}
I20260619 18:45:31.201587 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:31.200609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:35491)
0619 18:45:31.201244 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:31.201367 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":364,"threads_started":1}
I20260619 18:45:31.671610 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:31.670783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:36263)
0619 18:45:31.671336 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:31.671439 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260619 18:45:32.184710 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:32.183908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:52693)
0619 18:45:32.184417 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:32.184525 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":270,"threads_started":1}
I20260619 18:45:32.203693 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:32.202689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:36777)
0619 18:45:32.203199 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:32.203292 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260619 18:45:32.673386 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:32.672604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:37049)
0619 18:45:32.673092 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:32.673201 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260619 18:45:33.189484 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:33.188550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:52567)
0619 18:45:33.189123 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:33.189212 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":276,"threads_started":1}
I20260619 18:45:33.205864 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:33.205025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:46645)
0619 18:45:33.205553 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:33.205661 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260619 18:45:33.675238 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:33.674425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:38703)
0619 18:45:33.674976 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:33.675068 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":238,"threads_started":1}
I20260619 18:45:34.191488 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:34.190672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:56567)
0619 18:45:34.191186 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:34.191300 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
I20260619 18:45:34.207871 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:34.207070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:57319)
0619 18:45:34.207578 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:34.207676 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
I20260619 18:45:34.676980 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:34.676211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:46193)
0619 18:45:34.676688 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:34.676786 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
I20260619 18:45:35.193316 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:35.192556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:52919)
0619 18:45:35.193019 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:35.193125 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260619 18:45:35.209815 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:35.209013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:49579)
0619 18:45:35.209514 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:35.209615 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":245,"threads_started":1}
I20260619 18:45:35.678968 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:35.677999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:55825)
0619 18:45:35.678618 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:35.678754 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":261,"threads_started":1}
I20260619 18:45:36.195466 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:36.194685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:58865)
0619 18:45:36.195138 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:36.195266 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260619 18:45:36.211642 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:36.210904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:58711)
0619 18:45:36.211349 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:36.211451 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260619 18:45:36.681349 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:36.680346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.130:59747)
0619 18:45:36.681020 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:36.681156 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":265,"threads_started":1}
I20260619 18:45:37.196666 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20040
I20260619 18:45:37.198027 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:37.196712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.129:58393)
0619 18:45:37.197661 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:37.197807 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":634,"threads_started":1}
I20260619 18:45:37.213901 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:37.212909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:32829 (local address 127.3.85.131:46153)
0619 18:45:37.213574 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:37.213713 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.190:32829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":375,"threads_started":1}
I20260619 18:45:37.223259 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:37.198231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.3.85.129:42503)
0619 18:45:37.199169 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:37.199194 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:37.199253 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:45:37.201127 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:37.201139 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:37.201609 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:37.201959 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:37.203641 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.204966 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.206601 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.207452 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.207626 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:37.216106 (+ 8480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:37.216116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:37.216137 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:37.216192 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:37.219613 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.220465 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.220473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.220481 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.220666 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.221106 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.221110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.221112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.221745 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:37.222011 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:37.222542 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 18:45:37.222952 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":529,"threads_started":1}
I20260619 18:45:37.224831 20203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:37.311873 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20040
I20260619 18:45:37.330505 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20211
I20260619 18:45:37.349665 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:37.331476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.3.85.130:33733)
0619 18:45:37.332054 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:37.332069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:37.332107 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:45:37.333794 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:37.333810 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:37.334284 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:37.334795 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:37.336315 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.337588 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.339006 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.339756 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.339997 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:37.342819 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:37.342828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:37.342846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:37.342895 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:37.346060 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.347042 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.347050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.347058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.347251 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.347672 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.347675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.347677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.348245 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:37.348501 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:37.348993 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 18:45:37.349372 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260619 18:45:37.351234 20334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:37.436254 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20211
I20260619 18:45:37.455039 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20342
I20260619 18:45:37.475510 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:37.456167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.3.85.131:42927)
0619 18:45:37.456817 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:37.456832 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:37.456876 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:45:37.458717 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:37.458729 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:37.459272 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:37.459608 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:37.461311 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.462700 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.464167 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:37.464966 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:37.465129 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:37.468241 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:37.468249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:37.468268 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:37.468321 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:37.471872 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.472650 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.472657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.472664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.472858 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:37.473284 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:37.473289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:37.473291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:37.473941 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:37.474182 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:37.474712 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 18:45:37.475195 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":330,"threads_started":1}
I20260619 18:45:37.477053 20466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:37.562083 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20342
2026-06-19T18:45:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65144 ms)
[----------] 30 tests from SecurityITest (307873 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:37 dist-test-slave-09fp krb5kdc[20528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:37 dist-test-slave-09fp krb5kdc[20528](info): set up 2 sockets
Jun 19 18:45:37 dist-test-slave-09fp krb5kdc[20528](info): commencing operation
krb5kdc: starting...
W20260619 18:45:38.678654 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:45:38 dist-test-slave-09fp krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:45:38.947588 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37559
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:39.294234 20541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:39.294732 20541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:39.294916 20541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:39.302820 20541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:39.302990 20541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:39.303104 20541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:39.303213 20541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:39.303303 20541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:39.313856 20541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:39.315415 20541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:39.317167 20541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:39.330389 20546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:39.330376 20547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:39.330684 20541 server_base.cc:1034] running on GCE node
W20260619 18:45:39.330389 20549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:39.332075 20541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:39.333683 20541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:39.334942 20541 hybrid_clock.cc:648] HybridClock initialized: now 1781894739334875 us; error 61 us; skew 500 ppm
I20260619 18:45:39.335366 20541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:39 dist-test-slave-09fp krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:39.338845 20541 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:45:39.340548 20541 webserver.cc:460] Webserver started at http://127.3.85.190:38929/ using document root <none> and password file <none>
I20260619 18:45:39.341320 20541 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:39.341490 20541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:39.341826 20541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:39.345201 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "8ea26003bb164872bfea4f69fb1b316a"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "b0b300ca8094244a8936c5f9b1252ca6"
server_key_iv: "6842aba414a2fcfa69a60149bf4e50f4"
server_key_version: "encryptionkey@0"
I20260619 18:45:39.346087 20541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "8ea26003bb164872bfea4f69fb1b316a"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "b0b300ca8094244a8936c5f9b1252ca6"
server_key_iv: "6842aba414a2fcfa69a60149bf4e50f4"
server_key_version: "encryptionkey@0"
I20260619 18:45:39.352092 20541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:45:39.356246 20556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:39.357815 20541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:39.358031 20541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "8ea26003bb164872bfea4f69fb1b316a"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "b0b300ca8094244a8936c5f9b1252ca6"
server_key_iv: "6842aba414a2fcfa69a60149bf4e50f4"
server_key_version: "encryptionkey@0"
I20260619 18:45:39.358263 20541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:39.388293 20541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:39.393132 20541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:39.393486 20541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:39.413867 20541 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37559
I20260619 18:45:39.413903 20607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37559 every 8 connection(s)
I20260619 18:45:39.415402 20541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:39.419567 20608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:39.424167 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20541
I20260619 18:45:39.424405 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:39.425037 3414 external_mini_cluster.cc:1428] Setting key 9a992ae0aabe0e60a31cefd39b0f068c
I20260619 18:45:39.432932 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: Bootstrap starting.
I20260619 18:45:39.436954 20608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:39.438421 20608 log.cc:824] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:39.441967 20608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: No bootstrap required, opened a new log
Jun 19 18:45:39 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:39.453282 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:39.427248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.3.85.190:37559)
0619 18:45:39.428177 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:39.428195 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:39.428254 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:45:39.430978 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:39.431015 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:39.431625 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:39.432151 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:39.433646 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:39.435754 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:39.437264 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:39.438384 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:39.438619 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:39.443318 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:39.443330 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:39.443352 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:39.443403 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:39.447865 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:39.448828 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:39.448843 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:39.448853 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:39.449125 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:39.449545 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:39.449550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:39.449554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:39.450403 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:39.450745 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:39.451192 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 18:45:39.451734 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":402,"threads_started":1}
I20260619 18:45:39.453409 20608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } }
I20260619 18:45:39.453887 20608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:39.454017 20608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea26003bb164872bfea4f69fb1b316a, State: Initialized, Role: FOLLOWER
I20260619 18:45:39.454779 20608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } }
I20260619 18:45:39.455097 20608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:39.455264 20608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:39.455467 20608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:39.458066 20608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } }
I20260619 18:45:39.458678 20608 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea26003bb164872bfea4f69fb1b316a; no voters:
I20260619 18:45:39.459282 20608 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:39.459432 20613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:39.460182 20613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [term 1 LEADER]: Becoming Leader. State: Replica: 8ea26003bb164872bfea4f69fb1b316a, State: Running, Role: LEADER
I20260619 18:45:39.460775 20613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } }
I20260619 18:45:39.461584 20608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:39.463949 20614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea26003bb164872bfea4f69fb1b316a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } } }
I20260619 18:45:39.464335 20614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:39.465333 20618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:39.465101 20615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea26003bb164872bfea4f69fb1b316a. Latest consensus state: current_term: 1 leader_uuid: "8ea26003bb164872bfea4f69fb1b316a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea26003bb164872bfea4f69fb1b316a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37559 } } }
I20260619 18:45:39.466483 20615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:39.471563 20618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:39.481937 20618 catalog_manager.cc:1347] Generated new cluster ID: a48322ea8b104c378d36214e5d55cfcf
I20260619 18:45:39.482105 20618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:39.506270 20618 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:39.508332 20618 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:39.520731 20618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: Generated new TSK 0
I20260619 18:45:39.521924 20618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:45:39.586634 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:39.948586 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:39.948967 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:39.949146 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:39.957245 20634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:39.957439 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:39.957626 20634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:39.968346 20634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:39.969797 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:39.971378 20634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:39.985857 20640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:39.985826 20639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:39.986123 20642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:39.986720 20634 server_base.cc:1034] running on GCE node
I20260619 18:45:39.987452 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:39.988514 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:39.989797 20634 hybrid_clock.cc:648] HybridClock initialized: now 1781894739989748 us; error 63 us; skew 500 ppm
I20260619 18:45:39.990257 20634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:39 dist-test-slave-09fp krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:39.993798 20634 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:45:39.995476 20634 webserver.cc:460] Webserver started at http://127.3.85.129:41659/ using document root <none> and password file <none>
I20260619 18:45:39.996361 20634 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:39.996526 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:39.996861 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:40.000170 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "f9860d5b5502434289dac8271eda7aac"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "a3ea8cc7f0158b44c20cd5289d96e7d9"
server_key_iv: "589b465067f437b112b97310b87189da"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.001070 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "f9860d5b5502434289dac8271eda7aac"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "a3ea8cc7f0158b44c20cd5289d96e7d9"
server_key_iv: "589b465067f437b112b97310b87189da"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.007133 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:45:40.011401 20649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.013018 20634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:40.013271 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "f9860d5b5502434289dac8271eda7aac"
format_stamp: "Formatted at 2026-06-19 18:45:39 on dist-test-slave-09fp"
server_key: "a3ea8cc7f0158b44c20cd5289d96e7d9"
server_key_iv: "589b465067f437b112b97310b87189da"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.013536 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:40.031522 20634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:40.036495 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:40.036826 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:40.037921 20634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:40.039472 20634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:40.039582 20634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.039747 20634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:40.039821 20634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.075296 20634 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:44947
I20260619 18:45:40.075345 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:44947 every 8 connection(s)
I20260619 18:45:40.076675 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:40.086171 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20634
I20260619 18:45:40.086392 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:40.087038 3414 external_mini_cluster.cc:1428] Setting key 89c0a6edda3fa16ee826ff02b7bccdf3
Jun 19 18:45:40 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:40.099615 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:40.079220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:46027 (local address 127.3.85.190:37559)
0619 18:45:40.079693 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:40.079698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:40.080528 (+ 830us) server_negotiation.cc:407] Connection header received
0619 18:45:40.082431 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:40.082437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:40.082579 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:40.082714 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:40.084323 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.085057 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.086859 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.087347 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.087495 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:40.091683 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:40.091689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:40.091692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:40.091729 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:40.094427 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:40.095372 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:40.095378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:40.095382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:40.095531 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:40.096160 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:40.096194 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:40.096201 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:40.096624 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:40.096811 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:40.098778 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0619 18:45:40.099142 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260619 18:45:40.100281 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:40.079651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37559 (local address 127.3.85.129:46027)
0619 18:45:40.080359 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:40.080397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:40.082132 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:40.082970 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:40.082980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:40.083521 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:40.084110 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:40.084129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.085257 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.085262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:40.086696 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:40.086710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.087615 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.087624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:40.087935 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:40.088752 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:40.088792 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:40.091400 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:40.094673 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:40.094685 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:40.094710 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:40.095218 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:40.095679 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:40.095685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:40.095689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:40.095973 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:40.096819 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:40.096832 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:40.097463 (+ 631us) client_negotiation.cc:769] Sending connection context
0619 18:45:40.097893 (+ 430us) client_negotiation.cc:240] Negotiation successful
0619 18:45:40.098665 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":276,"threads_started":1}
I20260619 18:45:40.102439 20762 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37559
I20260619 18:45:40.102905 20762 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:40.104022 20762 heartbeater.cc:511] Master 127.3.85.190:37559 requested a full tablet report, sending...
I20260619 18:45:40.106509 20573 ts_manager.cc:194] Registered new tserver with Master: f9860d5b5502434289dac8271eda7aac (127.3.85.129:44947)
I20260619 18:45:40.108080 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:46027
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:45:40.146056 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:40.489449 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:40.489814 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:40.489989 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:40.497602 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:40.497783 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:40.497977 20768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:40.508148 20768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:40.509711 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:40.511176 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:40.525669 20773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:40.525786 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:40.525760 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:40.526705 20768 server_base.cc:1034] running on GCE node
I20260619 18:45:40.527379 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:40.528417 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:40.529677 20768 hybrid_clock.cc:648] HybridClock initialized: now 1781894740529622 us; error 49 us; skew 500 ppm
I20260619 18:45:40.530020 20768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:40 dist-test-slave-09fp krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:40.533383 20768 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:45:40.535136 20768 webserver.cc:460] Webserver started at http://127.3.85.130:36661/ using document root <none> and password file <none>
I20260619 18:45:40.535904 20768 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:40.536060 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:40.536399 20768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:40.539712 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "62c7befb08c449039bbc6bd8fcef74fb"
format_stamp: "Formatted at 2026-06-19 18:45:40 on dist-test-slave-09fp"
server_key: "6370c95737a2c75d67b671e7e8ba89d9"
server_key_iv: "0e4df2d5d91dd6ef6c049dd3f8f17bc6"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.540628 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "62c7befb08c449039bbc6bd8fcef74fb"
format_stamp: "Formatted at 2026-06-19 18:45:40 on dist-test-slave-09fp"
server_key: "6370c95737a2c75d67b671e7e8ba89d9"
server_key_iv: "0e4df2d5d91dd6ef6c049dd3f8f17bc6"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.546345 20768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 18:45:40.550258 20783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.551891 20768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:45:40.552119 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "62c7befb08c449039bbc6bd8fcef74fb"
format_stamp: "Formatted at 2026-06-19 18:45:40 on dist-test-slave-09fp"
server_key: "6370c95737a2c75d67b671e7e8ba89d9"
server_key_iv: "0e4df2d5d91dd6ef6c049dd3f8f17bc6"
server_key_version: "encryptionkey@0"
I20260619 18:45:40.552348 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:40.570022 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:40.575116 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:40.575438 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:40.576478 20768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:40.577893 20768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:40.578006 20768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.578169 20768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:40.578256 20768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:40.613149 20768 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40433
I20260619 18:45:40.613193 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40433 every 8 connection(s)
I20260619 18:45:40.614606 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:40.621486 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20768
I20260619 18:45:40.621678 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:40.622162 3414 external_mini_cluster.cc:1428] Setting key 495ae37d1d88ed774d9c5bcdc290a3f3
Jun 19 18:45:40 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894740, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:40.637670 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:40.617385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51897 (local address 127.3.85.190:37559)
0619 18:45:40.617859 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:40.617864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:40.619038 (+ 1174us) server_negotiation.cc:407] Connection header received
0619 18:45:40.620978 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:40.620984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:40.621085 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:40.621213 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:40.623085 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.624199 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.625914 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.626363 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.626552 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:40.630297 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:40.630303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:40.630306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:40.630347 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:40.633672 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:40.634842 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:40.634848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:40.634851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:40.634999 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:40.635634 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:40.635639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:40.635641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:40.635941 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:40.636077 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:40.637160 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 18:45:40.637377 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260619 18:45:40.639240 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:40.618018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37559 (local address 127.3.85.130:51897)
0619 18:45:40.618858 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:40.618904 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:40.620672 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:40.621466 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:40.621474 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:40.622137 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:40.622866 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:40.622886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.624389 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.624393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:40.625725 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:40.625739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:40.626635 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:40.626643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:40.626925 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:40.627667 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:40.627690 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:40.629991 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:40.633881 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:40.633894 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:40.633917 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:40.634671 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:40.635138 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:40.635146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:40.635152 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:40.635447 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:40.636095 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:40.636115 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:40.636660 (+ 545us) client_negotiation.cc:769] Sending connection context
0619 18:45:40.636997 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 18:45:40.637641 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":384,"threads_started":1}
I20260619 18:45:40.640951 20896 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37559
I20260619 18:45:40.641316 20896 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:40.642118 20896 heartbeater.cc:511] Master 127.3.85.190:37559 requested a full tablet report, sending...
I20260619 18:45:40.643761 20573 ts_manager.cc:194] Registered new tserver with Master: 62c7befb08c449039bbc6bd8fcef74fb (127.3.85.130:40433)
I20260619 18:45:40.644667 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:51897
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:45:40.682163 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:41.031920 20902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:41.032307 20902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:41.032490 20902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:41.040313 20902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:41.040498 20902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:41.040699 20902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:41.051101 20902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:41.052531 20902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:41.053920 20902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:41.068890 20907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:41.068849 20910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:41.068814 20908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:41.069792 20902 server_base.cc:1034] running on GCE node
I20260619 18:45:41.070597 20902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:41.071709 20902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:41.072952 20902 hybrid_clock.cc:648] HybridClock initialized: now 1781894741072911 us; error 55 us; skew 500 ppm
I20260619 18:45:41.073315 20902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:41 dist-test-slave-09fp krb5kdc[20528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:41.079562 20902 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:45:41.081159 20902 webserver.cc:460] Webserver started at http://127.3.85.131:33915/ using document root <none> and password file <none>
I20260619 18:45:41.081933 20902 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:41.082095 20902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:41.082423 20902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:41.085575 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "518cd37938b249a4a54248000c5b5e6c"
format_stamp: "Formatted at 2026-06-19 18:45:41 on dist-test-slave-09fp"
server_key: "4c6dc914399ea6abe87cb52c9ad4a947"
server_key_iv: "c7852ba49e223e69c94f91523efa641e"
server_key_version: "encryptionkey@0"
I20260619 18:45:41.086558 20902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "518cd37938b249a4a54248000c5b5e6c"
format_stamp: "Formatted at 2026-06-19 18:45:41 on dist-test-slave-09fp"
server_key: "4c6dc914399ea6abe87cb52c9ad4a947"
server_key_iv: "c7852ba49e223e69c94f91523efa641e"
server_key_version: "encryptionkey@0"
I20260619 18:45:41.092291 20902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:45:41.096225 20917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:41.097703 20902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:45:41.097913 20902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "518cd37938b249a4a54248000c5b5e6c"
format_stamp: "Formatted at 2026-06-19 18:45:41 on dist-test-slave-09fp"
server_key: "4c6dc914399ea6abe87cb52c9ad4a947"
server_key_iv: "c7852ba49e223e69c94f91523efa641e"
server_key_version: "encryptionkey@0"
I20260619 18:45:41.098147 20902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:41.111661 20762 heartbeater.cc:503] Master 127.3.85.190:37559 was elected leader, sending a full tablet report...
I20260619 18:45:41.122088 20902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:41.127061 20902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:41.127441 20902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:41.128585 20902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:41.129994 20902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:41.130070 20902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:41.130167 20902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:41.130211 20902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:41.165693 20902 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:46429
I20260619 18:45:41.165737 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:46429 every 8 connection(s)
I20260619 18:45:41.167188 20902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:41.168995 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 20902
I20260619 18:45:41.169175 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:41.169653 3414 external_mini_cluster.cc:1428] Setting key 6647e33e13b48c81c2569f06b0fe836d
Jun 19 18:45:41 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894741, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:41.191500 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.172322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:45025 (local address 127.3.85.190:37559)
0619 18:45:41.172787 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.172791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.174305 (+ 1514us) server_negotiation.cc:407] Connection header received
0619 18:45:41.176476 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.176480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.176557 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.176658 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:41.178420 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.179301 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.180945 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.181333 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.181483 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.185440 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:41.185451 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:41.185455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:41.185492 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:41.188446 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.189251 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.189255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.189258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.189381 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.189809 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.189813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.189815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.190097 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:41.190220 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.191020 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.191215 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":1021,"threads_started":1}
I20260619 18:45:41.192936 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.173135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37559 (local address 127.3.85.131:45025)
0619 18:45:41.174126 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.174176 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.176176 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.176917 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.176926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.177542 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:41.178197 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.178218 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.179476 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.179480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.180798 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.180812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.181592 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.181598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.181841 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.182591 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:41.182617 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:41.185191 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:41.188628 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:41.188636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:41.188651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:41.189139 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:41.189490 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:41.189493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:41.189495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:41.189711 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:41.190217 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:41.190223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:41.190686 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.190937 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.191513 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":430,"threads_started":1}
I20260619 18:45:41.194382 21030 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37559
I20260619 18:45:41.194752 21030 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:41.195506 21030 heartbeater.cc:511] Master 127.3.85.190:37559 requested a full tablet report, sending...
I20260619 18:45:41.197173 20573 ts_manager.cc:194] Registered new tserver with Master: 518cd37938b249a4a54248000c5b5e6c (127.3.85.131:46429)
I20260619 18:45:41.198401 20573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:45025
I20260619 18:45:41.199285 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:41.220137 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.204393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.3.85.190:37559)
0619 18:45:41.204672 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.204679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.204912 (+ 233us) server_negotiation.cc:407] Connection header received
0619 18:45:41.205148 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.205152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.205236 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.205330 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:41.207364 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.208157 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.209811 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.210428 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.210594 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.213762 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:41.213768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:41.213771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:41.213814 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:41.216793 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.217715 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.217725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.217731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.217914 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.218362 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.218366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.218368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.218838 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:41.219069 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.219533 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.219745 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 18:45:41.225595 20572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:41.233223 20572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:41.268232 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.249983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.3.85.130:40433)
0619 18:45:41.258617 (+ 8634us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.258631 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.258669 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:45:41.258798 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.258808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.259132 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.259376 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.260241 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.262181 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.265279 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.266841 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.266995 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.267222 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.267681 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.267873 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8423,"thread_start_us":8305,"threads_started":1}
I20260619 18:45:41.268710 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.249500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40433 (local address 127.0.0.1:40858)
0619 18:45:41.258273 (+ 8773us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.258296 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.258413 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.259416 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.259423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.259457 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.259986 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.260001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.262361 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.262367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.264929 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.264958 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.267079 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.267086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.267319 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.267376 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.267545 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.267753 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8640,"mutex_wait_us":45,"thread_start_us":8501,"threads_started":1}
I20260619 18:45:41.269436 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.247105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46429 (local address 127.0.0.1:43192)
0619 18:45:41.257447 (+ 10342us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.257474 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.257680 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.260335 (+ 2655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.260341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.260364 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.260775 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.260786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.263097 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.263102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.266364 (+ 3262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.266383 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.268457 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.268465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.268655 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.268705 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.268781 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.269026 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10169,"mutex_wait_us":228,"thread_start_us":1394,"threads_started":1}
I20260619 18:45:41.269840 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.247720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.3.85.131:46429)
0619 18:45:41.259023 (+ 11303us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.259030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.259061 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:45:41.259857 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.259866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.260093 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.260930 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.261172 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.262918 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.266639 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.268194 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.268949 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.269234 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.269316 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.269512 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11135,"thread_start_us":10976,"threads_started":1}
I20260619 18:45:41.269867 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.247223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.3.85.129:44947)
0619 18:45:41.259124 (+ 11901us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.259133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.259166 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:45:41.259261 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.259267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.259522 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.260631 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.260905 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.262820 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.266593 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.268281 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.268995 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.269252 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.269329 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.269524 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11705,"spinlock_wait_cycles":27520,"thread_start_us":11680,"threads_started":1}
I20260619 18:45:41.270356 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.246903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49058)
0619 18:45:41.249034 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.249098 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.249331 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.259812 (+ 10481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.259824 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.259856 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.260353 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.260368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.262982 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.262989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.266370 (+ 3381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.266383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.268562 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.268569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.268727 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.268752 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.268808 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.269859 (+ 1051us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1782,"mutex_wait_us":1080,"thread_start_us":10213,"threads_started":1}
I20260619 18:45:41.273561 20965 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4fe9275fb6445bb336c678f124876f (DEFAULT_TABLE table=test-table [id=35f3bec0cb5b441b8a45c1dc4aadb901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:41.273659 20831 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4fe9275fb6445bb336c678f124876f (DEFAULT_TABLE table=test-table [id=35f3bec0cb5b441b8a45c1dc4aadb901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:41.274111 20697 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4fe9275fb6445bb336c678f124876f (DEFAULT_TABLE table=test-table [id=35f3bec0cb5b441b8a45c1dc4aadb901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:41.275184 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4fe9275fb6445bb336c678f124876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:41.275226 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4fe9275fb6445bb336c678f124876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:41.275668 20697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4fe9275fb6445bb336c678f124876f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:41.287528 21049 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Bootstrap starting.
I20260619 18:45:41.290997 21051 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Bootstrap starting.
I20260619 18:45:41.293105 21049 tablet_bootstrap.cc:654] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:41.293103 21050 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Bootstrap starting.
I20260619 18:45:41.294739 21049 log.cc:824] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:41.295069 21051 tablet_bootstrap.cc:654] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:41.296223 21051 log.cc:824] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:41.296715 21050 tablet_bootstrap.cc:654] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:41.298041 21050 log.cc:824] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:41.298545 21049 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: No bootstrap required, opened a new log
I20260619 18:45:41.298858 21049 ts_tablet_manager.cc:1397] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260619 18:45:41.300657 21051 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: No bootstrap required, opened a new log
I20260619 18:45:41.300962 21051 ts_tablet_manager.cc:1397] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:45:41.301630 21050 tablet_bootstrap.cc:492] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: No bootstrap required, opened a new log
I20260619 18:45:41.301915 21050 ts_tablet_manager.cc:1397] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:45:41.309859 21050 raft_consensus.cc:348] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.310309 21050 raft_consensus.cc:374] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:41.310415 21050 raft_consensus.cc:729] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9860d5b5502434289dac8271eda7aac, State: Initialized, Role: FOLLOWER
I20260619 18:45:41.310693 21049 raft_consensus.cc:348] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.311159 21049 raft_consensus.cc:374] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:41.311261 21049 raft_consensus.cc:729] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518cd37938b249a4a54248000c5b5e6c, State: Initialized, Role: FOLLOWER
I20260619 18:45:41.311152 21050 consensus_queue.cc:260] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.312012 21049 consensus_queue.cc:260] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.313207 21050 ts_tablet_manager.cc:1428] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 18:45:41.312016 21051 raft_consensus.cc:348] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.314312 21030 heartbeater.cc:503] Master 127.3.85.190:37559 was elected leader, sending a full tablet report...
I20260619 18:45:41.314349 21051 raft_consensus.cc:374] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:41.314522 21051 raft_consensus.cc:729] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c7befb08c449039bbc6bd8fcef74fb, State: Initialized, Role: FOLLOWER
I20260619 18:45:41.315124 21049 ts_tablet_manager.cc:1428] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260619 18:45:41.315292 21051 consensus_queue.cc:260] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.317220 20896 heartbeater.cc:503] Master 127.3.85.190:37559 was elected leader, sending a full tablet report...
I20260619 18:45:41.318097 21051 ts_tablet_manager.cc:1428] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260619 18:45:41.329218 20763 tablet.cc:2369] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:41.366796 20897 tablet.cc:2369] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:41.419081 21031 tablet.cc:2369] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:41.494617 21056 raft_consensus.cc:482] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:41.494920 21056 raft_consensus.cc:504] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.496775 21056 leader_election.cc:288] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9860d5b5502434289dac8271eda7aac (127.3.85.129:44947), 62c7befb08c449039bbc6bd8fcef74fb (127.3.85.130:40433)
I20260619 18:45:41.497877 21055 raft_consensus.cc:482] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:41.498291 21055 raft_consensus.cc:504] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.500553 21055 leader_election.cc:288] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518cd37938b249a4a54248000c5b5e6c (127.3.85.131:46429), 62c7befb08c449039bbc6bd8fcef74fb (127.3.85.130:40433)
I20260619 18:45:41.506014 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.497318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40433 (local address 127.3.85.131:58489)
0619 18:45:41.498186 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.498204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.498331 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.498736 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.498740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.498760 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.499150 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.499162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.500672 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.500680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.503822 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.503840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.505073 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.505080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.505275 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.505323 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.505419 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.505578 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":1,"thread_start_us":499,"threads_started":1}
I20260619 18:45:41.506086 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.497279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:58489 (local address 127.3.85.130:40433)
0619 18:45:41.497571 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.497578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.498319 (+ 741us) server_negotiation.cc:407] Connection header received
0619 18:45:41.498469 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.498475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.498594 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.498705 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.499357 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.500435 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.504012 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.504839 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.505055 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.505135 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.505476 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.505695 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260619 18:45:41.507531 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.497230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.3.85.131:48575)
0619 18:45:41.497654 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.497705 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.497885 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.498553 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.498557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.498579 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.498772 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.498784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.500047 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.500053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.502262 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.502279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.506676 (+ 4397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.506685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.506917 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.506953 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.507051 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.507206 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":8}
I20260619 18:45:41.507308 20851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "518cd37938b249a4a54248000c5b5e6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7befb08c449039bbc6bd8fcef74fb" is_pre_election: true
I20260619 18:45:41.507979 20851 raft_consensus.cc:2454] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 518cd37938b249a4a54248000c5b5e6c in term 0.
I20260619 18:45:41.508888 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.497268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:48575 (local address 127.3.85.129:44947)
0619 18:45:41.497497 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.497504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.498041 (+ 537us) server_negotiation.cc:407] Connection header received
0619 18:45:41.498132 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.498137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.498234 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.498328 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.498982 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.499877 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.502538 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.506280 (+ 3742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.506480 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.506577 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.508331 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.508549 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260619 18:45:41.508862 20921 leader_election.cc:302] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 518cd37938b249a4a54248000c5b5e6c, 62c7befb08c449039bbc6bd8fcef74fb; no voters:
I20260619 18:45:41.509408 21056 raft_consensus.cc:2781] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:41.509573 21056 raft_consensus.cc:482] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:41.509689 21056 raft_consensus.cc:3037] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:41.509972 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "518cd37938b249a4a54248000c5b5e6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9860d5b5502434289dac8271eda7aac" is_pre_election: true
I20260619 18:45:41.510535 20717 raft_consensus.cc:2454] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 518cd37938b249a4a54248000c5b5e6c in term 0.
I20260619 18:45:41.512852 21056 raft_consensus.cc:504] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.513870 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.501075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:47455 (local address 127.3.85.131:46429)
0619 18:45:41.501286 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.501292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.501976 (+ 684us) server_negotiation.cc:407] Connection header received
0619 18:45:41.502344 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.502351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.502430 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.502526 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.507299 (+ 4773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.508432 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.511314 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.512733 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.512887 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.512980 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.513473 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.513611 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:45:41.513931 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.500839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:46429 (local address 127.3.85.129:47455)
0619 18:45:41.501833 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.501855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.502114 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.502741 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.502749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.502787 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.507119 (+ 4332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.507139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.508626 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.508642 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.511092 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.511107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.513038 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.513053 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.513288 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.513314 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.513430 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.513582 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":652,"threads_started":1}
I20260619 18:45:41.514555 20851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "518cd37938b249a4a54248000c5b5e6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7befb08c449039bbc6bd8fcef74fb"
I20260619 18:45:41.514611 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "518cd37938b249a4a54248000c5b5e6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9860d5b5502434289dac8271eda7aac"
I20260619 18:45:41.514804 20851 raft_consensus.cc:3037] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:41.514853 20717 raft_consensus.cc:3037] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:41.515012 20985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "f9860d5b5502434289dac8271eda7aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518cd37938b249a4a54248000c5b5e6c" is_pre_election: true
I20260619 18:45:41.515568 20985 raft_consensus.cc:2379] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9860d5b5502434289dac8271eda7aac in current term 1: Already voted for candidate 518cd37938b249a4a54248000c5b5e6c in this term.
I20260619 18:45:41.515753 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.501958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40433 (local address 127.3.85.129:35431)
0619 18:45:41.507121 (+ 5163us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.507141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:41.507278 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:41.508044 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:41.508051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:41.508080 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:41.508630 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.508642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.510551 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.510557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.512710 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:41.512728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.514763 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.514770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:41.514966 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.515003 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:45:41.515105 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:45:41.515296 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4981,"mutex_wait_us":71,"thread_start_us":4779,"threads_started":1}
I20260619 18:45:41.516494 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.502488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:35431 (local address 127.3.85.130:40433)
0619 18:45:41.503664 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.503673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.507387 (+ 3714us) server_negotiation.cc:407] Connection header received
0619 18:45:41.507462 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.507468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.507547 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.507742 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:41.508885 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.510327 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.512933 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.514274 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.514498 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.514597 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.515963 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.516132 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":732,"threads_started":1}
I20260619 18:45:41.517009 21056 leader_election.cc:288] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [CANDIDATE]: Term 1 election: Requested vote from peers f9860d5b5502434289dac8271eda7aac (127.3.85.129:44947), 62c7befb08c449039bbc6bd8fcef74fb (127.3.85.130:40433)
I20260619 18:45:41.517568 20850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4fe9275fb6445bb336c678f124876f" candidate_uuid: "f9860d5b5502434289dac8271eda7aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c7befb08c449039bbc6bd8fcef74fb" is_pre_election: true
I20260619 18:45:41.518404 20851 raft_consensus.cc:2454] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 518cd37938b249a4a54248000c5b5e6c in term 1.
I20260619 18:45:41.518746 20717 raft_consensus.cc:2454] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 518cd37938b249a4a54248000c5b5e6c in term 1.
I20260619 18:45:41.518651 20653 leader_election.cc:302] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9860d5b5502434289dac8271eda7aac; no voters: 518cd37938b249a4a54248000c5b5e6c, 62c7befb08c449039bbc6bd8fcef74fb
I20260619 18:45:41.518812 20921 leader_election.cc:302] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 518cd37938b249a4a54248000c5b5e6c, 62c7befb08c449039bbc6bd8fcef74fb; no voters:
I20260619 18:45:41.519151 21055 raft_consensus.cc:2726] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:45:41.519196 21056 raft_consensus.cc:2781] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:41.519716 21056 raft_consensus.cc:686] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 LEADER]: Becoming Leader. State: Replica: 518cd37938b249a4a54248000c5b5e6c, State: Running, Role: LEADER
I20260619 18:45:41.520236 21056 consensus_queue.cc:237] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } }
I20260619 18:45:41.525193 20573 catalog_manager.cc:5495] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 518cd37938b249a4a54248000c5b5e6c (127.3.85.131). New cstate: current_term: 1 leader_uuid: "518cd37938b249a4a54248000c5b5e6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "518cd37938b249a4a54248000c5b5e6c" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:41.558156 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.551847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.3.85.131:46429)
0619 18:45:41.552019 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.552023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.552150 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:45:41.552329 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.552332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.552437 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.552580 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:41.552966 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.553745 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.555724 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.556178 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.556282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.556962 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:41.557501 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:41.557630 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.557817 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.557901 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:45:41.567289 20851 raft_consensus.cc:1261] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 1 FOLLOWER]: Refusing update from remote peer 518cd37938b249a4a54248000c5b5e6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:41.567525 20717 raft_consensus.cc:1261] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 1 FOLLOWER]: Refusing update from remote peer 518cd37938b249a4a54248000c5b5e6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:41.568291 21062 consensus_queue.cc:1035] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c7befb08c449039bbc6bd8fcef74fb" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:41.568696 21056 consensus_queue.cc:1035] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9860d5b5502434289dac8271eda7aac" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:41.600545 20573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:41.600912 20573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:41.605252 20573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea26003bb164872bfea4f69fb1b316a: Sending DeleteTablet for 3 replicas of tablet 8e4fe9275fb6445bb336c678f124876f
I20260619 18:45:41.606632 20697 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4fe9275fb6445bb336c678f124876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:41 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:49058
I20260619 18:45:41.606814 20831 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4fe9275fb6445bb336c678f124876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:41 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40858
I20260619 18:45:41.607038 20965 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4fe9275fb6445bb336c678f124876f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:41 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:43192
I20260619 18:45:41.607764 21074 tablet_replica.cc:330] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: stopping tablet replica
I20260619 18:45:41.608383 21074 raft_consensus.cc:2229] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:41.608685 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20634
I20260619 18:45:41.609066 21074 raft_consensus.cc:2258] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:41.614785 21074 ts_tablet_manager.cc:1905] T 8e4fe9275fb6445bb336c678f124876f P f9860d5b5502434289dac8271eda7aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:41.618196 21075 tablet_replica.cc:330] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: stopping tablet replica
I20260619 18:45:41.618929 21075 raft_consensus.cc:2229] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:41.619544 21075 raft_consensus.cc:2258] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:41.620350 21076 tablet_replica.cc:330] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: stopping tablet replica
I20260619 18:45:41.620805 21076 raft_consensus.cc:2229] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:41.621515 21076 raft_consensus.cc:2258] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 18:45:41 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:45:41.622494 21075 ts_tablet_manager.cc:1905] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:41.625108 21076 ts_tablet_manager.cc:1905] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:41.631606 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.609977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.3.85.129:44947)
0619 18:45:41.610194 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.610202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.610345 (+ 143us) server_negotiation.cc:407] Connection header received
0619 18:45:41.610705 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.610713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.610816 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.610932 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:41.613267 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.614644 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.616791 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.617557 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.617686 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.623554 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:41.623576 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:41.623596 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:41.623661 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:41.627848 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.628860 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.628870 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.628885 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.629047 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.629485 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.629491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.629494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.629983 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:41.630260 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.630936 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.631180 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:45:41.632561 21076 ts_tablet_manager.cc:1918] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:41.632752 21076 log.cc:1196] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/8e4fe9275fb6445bb336c678f124876f
I20260619 18:45:41.633184 20757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:41.633314 21076 ts_tablet_manager.cc:1939] T 8e4fe9275fb6445bb336c678f124876f P 518cd37938b249a4a54248000c5b5e6c: Deleting consensus metadata
I20260619 18:45:41.634559 21075 ts_tablet_manager.cc:1918] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:41.634778 21075 log.cc:1196] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/8e4fe9275fb6445bb336c678f124876f
I20260619 18:45:41.635576 21075 ts_tablet_manager.cc:1939] T 8e4fe9275fb6445bb336c678f124876f P 62c7befb08c449039bbc6bd8fcef74fb: Deleting consensus metadata
I20260619 18:45:41.636786 20560 catalog_manager.cc:4841] TS 518cd37938b249a4a54248000c5b5e6c (127.3.85.131:46429): tablet 8e4fe9275fb6445bb336c678f124876f (table test-table [id=35f3bec0cb5b441b8a45c1dc4aadb901]) successfully deleted
I20260619 18:45:41.639073 20560 catalog_manager.cc:4841] TS 62c7befb08c449039bbc6bd8fcef74fb (127.3.85.130:40433): tablet 8e4fe9275fb6445bb336c678f124876f (table test-table [id=35f3bec0cb5b441b8a45c1dc4aadb901]) successfully deleted
I20260619 18:45:41.757589 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20634
I20260619 18:45:41.776859 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20768
W20260619 18:45:41.776998 20558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:44947 (error 108)
I20260619 18:45:41.778947 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.778428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49072)
0619 18:45:41.778632 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.778759 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 18:45:41.779234 20558 catalog_manager.cc:4584] TS f9860d5b5502434289dac8271eda7aac (127.3.85.129:44947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e4fe9275fb6445bb336c678f124876f: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Jun 19 18:45:41 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:45:41.796059 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.778126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.3.85.130:40433)
0619 18:45:41.778366 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.778372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.778390 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:45:41.778504 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.778510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.778592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.778718 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:41.780557 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.781272 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.783348 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.783811 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.783950 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.789281 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:41.789327 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:41.789349 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:41.789436 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:41.792650 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.793527 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.793536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.793539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.793691 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.794115 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.794122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.794130 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.794767 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:41.795140 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.795415 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.795676 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:45:41.797106 20891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:41.833024 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.832421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49086)
0619 18:45:41.832693 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.832809 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 18:45:41.834365 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.833909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49094)
0619 18:45:41.834093 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.834197 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:45:41.894212 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20768
I20260619 18:45:41.895329 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.894744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49110)
0619 18:45:41.894984 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.895105 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:45:41.896709 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.896191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49120)
0619 18:45:41.896410 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.896537 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:45:41.913813 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20902
Jun 19 18:45:41 dist-test-slave-09fp krb5kdc[20528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:45:41.938855 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.914685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.3.85.131:46429)
0619 18:45:41.914879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:41.914886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:41.915165 (+ 279us) server_negotiation.cc:407] Connection header received
0619 18:45:41.915364 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:41.915368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:41.915462 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:41.915568 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:41.917428 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.918172 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.920145 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:41.920714 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:41.920861 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:41.932127 (+ 11266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:41.932144 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:41.932162 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:41.932238 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:41.935571 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.936437 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.936445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.936447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.936605 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:41.937054 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:41.937057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:41.937059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:41.937499 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:41.937850 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:41.938192 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 18:45:41.938516 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:45:41.939615 21025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:41.978139 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.977458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49130)
0619 18:45:41.977764 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.977904 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 18:45:41.980008 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:41.979072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49138)
0619 18:45:41.979642 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:41.979770 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 18:45:42.086782 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20902
I20260619 18:45:42.107429 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 20541
I20260619 18:45:42.108628 20603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:42.220549 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:42.219813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:44947 (local address 127.0.0.1:49150)
0619 18:45:42.220249 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:42.220354 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:44947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260619 18:45:42.220530 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 20541
2026-06-19T18:45:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T18:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:42Z Disabled control of system clock
I20260619 18:45:42.301865 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39883
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:42.656257 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:42.656605 21086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:42.656692 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:42.664448 21086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:42.664620 21086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:42.664712 21086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:42.664820 21086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:42.664914 21086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:42.674723 21086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:42.676170 21086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:42.678118 21086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:42.691259 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:42.691438 21091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:42.691440 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:42.692380 21086 server_base.cc:1034] running on GCE node
I20260619 18:45:42.693320 21086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:42.694974 21086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:42.696228 21086 hybrid_clock.cc:648] HybridClock initialized: now 1781894742696191 us; error 63 us; skew 500 ppm
I20260619 18:45:42.696606 21086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:42.703542 21086 webserver.cc:460] Webserver started at http://127.3.85.190:45381/ using document root <none> and password file <none>
I20260619 18:45:42.704372 21086 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:42.704530 21086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:42.704881 21086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:42.709311 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "6c77b7e96d9449778e064d8b0d51f3f7"
format_stamp: "Formatted at 2026-06-19 18:45:42 on dist-test-slave-09fp"
server_key: "98e1a8f404e70dd5afb247029d53f51a"
server_key_iv: "765fff85775b2f5f511f77e985190d86"
server_key_version: "encryptionkey@0"
I20260619 18:45:42.710592 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "6c77b7e96d9449778e064d8b0d51f3f7"
format_stamp: "Formatted at 2026-06-19 18:45:42 on dist-test-slave-09fp"
server_key: "98e1a8f404e70dd5afb247029d53f51a"
server_key_iv: "765fff85775b2f5f511f77e985190d86"
server_key_version: "encryptionkey@0"
I20260619 18:45:42.718537 21086 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 18:45:42.722432 21100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:42.724089 21086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:42.724326 21086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "6c77b7e96d9449778e064d8b0d51f3f7"
format_stamp: "Formatted at 2026-06-19 18:45:42 on dist-test-slave-09fp"
server_key: "98e1a8f404e70dd5afb247029d53f51a"
server_key_iv: "765fff85775b2f5f511f77e985190d86"
server_key_version: "encryptionkey@0"
I20260619 18:45:42.724579 21086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:42.741525 21086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:42.742542 21086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:42.742846 21086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:42.763345 21086 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39883
I20260619 18:45:42.763386 21151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39883 every 8 connection(s)
I20260619 18:45:42.764837 21086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:42.769117 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21086
I20260619 18:45:42.769157 21152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:42.769326 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:42.769800 3414 external_mini_cluster.cc:1428] Setting key b2cb82de2ecd27ff85986d28b779df30
I20260619 18:45:42.787537 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: Bootstrap starting.
I20260619 18:45:42.787537 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:42.771930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.3.85.190:39883)
0619 18:45:42.773221 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:42.773239 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:42.773303 (+ 64us) server_negotiation.cc:407] Connection header received
0619 18:45:42.775478 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:42.775522 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:42.776356 (+ 834us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:42.776909 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:42.777599 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:42.779881 (+ 2282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:42.781566 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:42.782818 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:42.783045 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:42.784204 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:42.784216 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:42.784240 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:42.785111 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:42.785222 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:42.785255 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:42.785369 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:42.785670 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 18:45:42.786065 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":410,"threads_started":1}
I20260619 18:45:42.791292 21152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:42.792305 21152 log.cc:824] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:42.795392 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: No bootstrap required, opened a new log
I20260619 18:45:42.804683 21152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } }
I20260619 18:45:42.805160 21152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:42.805285 21152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c77b7e96d9449778e064d8b0d51f3f7, State: Initialized, Role: FOLLOWER
I20260619 18:45:42.805985 21152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } }
I20260619 18:45:42.806245 21152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:42.806399 21152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:42.806623 21152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:42.809087 21152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } }
I20260619 18:45:42.809698 21152 leader_election.cc:302] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c77b7e96d9449778e064d8b0d51f3f7; no voters:
I20260619 18:45:42.810423 21152 leader_election.cc:288] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:42.810738 21157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:42.811558 21157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6c77b7e96d9449778e064d8b0d51f3f7, State: Running, Role: LEADER
I20260619 18:45:42.812158 21157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } }
I20260619 18:45:42.812829 21152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:42.815395 21159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c77b7e96d9449778e064d8b0d51f3f7. Latest consensus state: current_term: 1 leader_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } } }
I20260619 18:45:42.815837 21159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:42.816542 21158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c77b7e96d9449778e064d8b0d51f3f7" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39883 } } }
I20260619 18:45:42.816964 21158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:42.819139 21162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:42.824950 21162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:42.835129 21162 catalog_manager.cc:1347] Generated new cluster ID: 080f1065a6204ec298e08d00323844b5
I20260619 18:45:42.835264 21162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:42.854337 21162 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:42.855793 21162 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:42.872884 21162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: Generated new TSK 0
I20260619 18:45:42.873911 21162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:45:42.881477 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--builtin_ntp_servers=127.3.85.148:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:43.279923 21176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:43.280259 21176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:43.280351 21176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:43.288287 21176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:43.288458 21176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:43.288630 21176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:43.298295 21176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:43.299721 21176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:43.301551 21176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:43.315373 21184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.315407 21182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.315393 21181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:43.316032 21176 server_base.cc:1034] running on GCE node
I20260619 18:45:43.316709 21176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:43.317873 21176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:43.319118 21176 hybrid_clock.cc:648] HybridClock initialized: now 1781894743319071 us; error 50 us; skew 500 ppm
I20260619 18:45:43.319470 21176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:43.321864 21176 webserver.cc:460] Webserver started at http://127.3.85.129:42437/ using document root <none> and password file <none>
I20260619 18:45:43.322706 21176 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:43.322829 21176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:43.323187 21176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:43.326270 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "7a7057f5bdfd44e38f391dbbd050435e"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "f1156d5f4357c8ce59970ab3fea97bdc"
server_key_iv: "86d12df67b5476729f1c7a3101d498ca"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.327267 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "7a7057f5bdfd44e38f391dbbd050435e"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "f1156d5f4357c8ce59970ab3fea97bdc"
server_key_iv: "86d12df67b5476729f1c7a3101d498ca"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.333109 21176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:45:43.337064 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.338653 21176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:43.338840 21176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "7a7057f5bdfd44e38f391dbbd050435e"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "f1156d5f4357c8ce59970ab3fea97bdc"
server_key_iv: "86d12df67b5476729f1c7a3101d498ca"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.339083 21176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:43.357942 21176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:43.358893 21176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:43.359239 21176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:43.360278 21176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:43.361904 21176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:43.362015 21176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.362176 21176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:43.362272 21176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.397131 21176 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:33407
I20260619 18:45:43.397184 21302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:33407 every 8 connection(s)
I20260619 18:45:43.398653 21176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:43.400892 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21176
I20260619 18:45:43.401094 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:43.401604 3414 external_mini_cluster.cc:1428] Setting key db3f4775697de2e473bd2099d48351f6
I20260619 18:45:43.404731 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--builtin_ntp_servers=127.3.85.148:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:43.413542 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:43.401814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:37227 (local address 127.3.85.190:39883)
0619 18:45:43.402394 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:43.402403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:43.403998 (+ 1595us) server_negotiation.cc:407] Connection header received
0619 18:45:43.406036 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:43.406042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:43.406141 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:43.406276 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:43.407493 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.408247 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.409940 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.410354 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.410537 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:43.412621 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:43.412627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:43.412631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:43.412653 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:43.412697 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:43.412712 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:43.412796 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:43.413110 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 18:45:43.413211 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":284,"threads_started":1}
I20260619 18:45:43.415218 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:43.402605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39883 (local address 127.3.85.129:37227)
0619 18:45:43.403842 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:43.403894 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:43.405725 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:43.406584 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:43.406594 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:43.406697 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:43.407310 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:43.407328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.408426 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.408438 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:43.409762 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:43.409777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.410661 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.410668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:43.410931 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:43.412031 (+ 1100us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:43.412038 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:43.412104 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:43.412237 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:43.412828 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:43.412843 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:43.412855 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:45:43.413070 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 18:45:43.413677 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":344,"threads_started":1}
I20260619 18:45:43.417085 21303 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39883
I20260619 18:45:43.417466 21303 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:43.418515 21303 heartbeater.cc:511] Master 127.3.85.190:39883 requested a full tablet report, sending...
I20260619 18:45:43.421066 21117 ts_manager.cc:194] Registered new tserver with Master: 7a7057f5bdfd44e38f391dbbd050435e (127.3.85.129:33407)
I20260619 18:45:43.422739 21117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.129:37227
W20260619 18:45:43.757390 21307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:43.757768 21307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:43.757937 21307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:43.767817 21307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:43.767971 21307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:43.768153 21307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:43.783782 21307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:43.785429 21307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:43.787573 21307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:43.802156 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.802240 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:43.802271 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:43.803352 21307 server_base.cc:1034] running on GCE node
I20260619 18:45:43.804179 21307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:43.805680 21307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:43.806948 21307 hybrid_clock.cc:648] HybridClock initialized: now 1781894743806878 us; error 70 us; skew 500 ppm
I20260619 18:45:43.807325 21307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:43.809911 21307 webserver.cc:460] Webserver started at http://127.3.85.130:43607/ using document root <none> and password file <none>
I20260619 18:45:43.810799 21307 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:43.810952 21307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:43.811305 21307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:43.814787 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "9f9b2d283b934cc9a4cea324cc1e00c9"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "b32797dc1011e22c05089d3492e6e075"
server_key_iv: "16438f580203565dac16e4786bb12e9a"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.815747 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "9f9b2d283b934cc9a4cea324cc1e00c9"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "b32797dc1011e22c05089d3492e6e075"
server_key_iv: "16438f580203565dac16e4786bb12e9a"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.821666 21307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 18:45:43.825673 21321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.827412 21307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:43.827621 21307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "9f9b2d283b934cc9a4cea324cc1e00c9"
format_stamp: "Formatted at 2026-06-19 18:45:43 on dist-test-slave-09fp"
server_key: "b32797dc1011e22c05089d3492e6e075"
server_key_iv: "16438f580203565dac16e4786bb12e9a"
server_key_version: "encryptionkey@0"
I20260619 18:45:43.827876 21307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:43.859809 21307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:43.860733 21307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:43.861066 21307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:43.862109 21307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:43.863843 21307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:43.863986 21307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.864146 21307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:43.864238 21307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:43.898759 21307 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:40495
I20260619 18:45:43.898846 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:40495 every 8 connection(s)
I20260619 18:45:43.900134 21307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:43.903929 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21307
I20260619 18:45:43.904131 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:43.904592 3414 external_mini_cluster.cc:1428] Setting key 990dbdf63a3bc8062f22b71eb8ccca5f
I20260619 18:45:43.907936 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--builtin_ntp_servers=127.3.85.148:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:43.915679 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:43.903051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:37553 (local address 127.3.85.190:39883)
0619 18:45:43.903236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:43.903240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:43.904946 (+ 1706us) server_negotiation.cc:407] Connection header received
0619 18:45:43.907217 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:43.907231 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:43.907325 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:43.907442 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:43.908775 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.910017 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.911945 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.912740 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.912974 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:43.914676 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:43.914685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:43.914693 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:43.914723 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:43.914767 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:43.914786 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:43.914882 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:43.915124 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:45:43.915257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:45:43.916651 21436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:43.903577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39883 (local address 127.3.85.130:37553)
0619 18:45:43.904746 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:43.904793 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:43.906909 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:43.907714 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:43.907724 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:43.907827 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:43.908568 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:43.908589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.910237 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.910243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:43.911752 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:43.911771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:43.913090 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:43.913097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:43.913328 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:43.914237 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:43.914243 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:43.914291 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:43.914387 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:43.914890 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:43.914899 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:43.914905 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:45:43.915063 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 18:45:43.915413 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":397,"threads_started":1}
I20260619 18:45:43.918311 21434 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39883
I20260619 18:45:43.918726 21434 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:43.919552 21434 heartbeater.cc:511] Master 127.3.85.190:39883 requested a full tablet report, sending...
I20260619 18:45:43.921306 21117 ts_manager.cc:194] Registered new tserver with Master: 9f9b2d283b934cc9a4cea324cc1e00c9 (127.3.85.130:40495)
I20260619 18:45:43.922602 21117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.130:37553
W20260619 18:45:44.261857 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:44.262176 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:44.262265 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:44.269888 21437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:44.270020 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:44.270133 21437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:44.279651 21437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:44.281001 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:44.283004 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:44.297658 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.297684 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:44.297739 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:44.298628 21437 server_base.cc:1034] running on GCE node
I20260619 18:45:44.299312 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:44.300374 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:44.301599 21437 hybrid_clock.cc:648] HybridClock initialized: now 1781894744301553 us; error 47 us; skew 500 ppm
I20260619 18:45:44.301968 21437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:44.304545 21437 webserver.cc:460] Webserver started at http://127.3.85.131:34153/ using document root <none> and password file <none>
I20260619 18:45:44.305289 21437 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:44.305385 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:44.305745 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:44.308770 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "b8543d621c0c4a22b0618bbc4ffd659b"
format_stamp: "Formatted at 2026-06-19 18:45:44 on dist-test-slave-09fp"
server_key: "ecbfac682408ddbadbe3787c261d189c"
server_key_iv: "2f60dc0c5d3765bdf641c0e7d017551e"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.309552 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "b8543d621c0c4a22b0618bbc4ffd659b"
format_stamp: "Formatted at 2026-06-19 18:45:44 on dist-test-slave-09fp"
server_key: "ecbfac682408ddbadbe3787c261d189c"
server_key_iv: "2f60dc0c5d3765bdf641c0e7d017551e"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.315384 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 18:45:44.319305 21451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.320873 21437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:44.321082 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "b8543d621c0c4a22b0618bbc4ffd659b"
format_stamp: "Formatted at 2026-06-19 18:45:44 on dist-test-slave-09fp"
server_key: "ecbfac682408ddbadbe3787c261d189c"
server_key_iv: "2f60dc0c5d3765bdf641c0e7d017551e"
server_key_version: "encryptionkey@0"
I20260619 18:45:44.321314 21437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:44.390251 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:44.391291 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:44.391628 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:44.393016 21437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:44.394810 21437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:44.394909 21437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.394997 21437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:44.395056 21437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:44.427213 21303 heartbeater.cc:503] Master 127.3.85.190:39883 was elected leader, sending a full tablet report...
I20260619 18:45:44.431893 21437 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:41861
I20260619 18:45:44.432005 21563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:41861 every 8 connection(s)
I20260619 18:45:44.433435 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:44.438964 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21437
I20260619 18:45:44.439147 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:44.439729 3414 external_mini_cluster.cc:1428] Setting key c69586420e22f790f1c952560c3732b6
I20260619 18:45:44.448508 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.436190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:51793 (local address 127.3.85.190:39883)
0619 18:45:44.436692 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.436697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.437837 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 18:45:44.440509 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.440516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.440620 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.440750 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.442132 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.443110 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.444981 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.445462 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.445587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.447557 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.447563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.447567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:44.447590 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.447636 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:44.447650 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.447737 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.448043 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.448158 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":281,"threads_started":1}
I20260619 18:45:44.449555 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.436661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39883 (local address 127.3.85.131:51793)
0619 18:45:44.437643 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.437683 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.440134 (+ 2451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.441030 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.441040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.441172 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:44.441943 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.441964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.443317 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.443321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.444791 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.444807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.445768 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.445775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.446146 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.447129 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:44.447132 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:44.447180 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:44.447291 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:44.447790 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:44.447800 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:44.447807 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.447999 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.448394 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260619 18:45:44.451326 21564 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39883
I20260619 18:45:44.451694 21564 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:44.452534 21564 heartbeater.cc:511] Master 127.3.85.190:39883 requested a full tablet report, sending...
I20260619 18:45:44.454380 21117 ts_manager.cc:194] Registered new tserver with Master: b8543d621c0c4a22b0618bbc4ffd659b (127.3.85.131:41861)
I20260619 18:45:44.455559 21117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.131:51793
I20260619 18:45:44.456514 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:44.471258 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.464872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.3.85.190:39883)
0619 18:45:44.465157 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.465165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.465462 (+ 297us) server_negotiation.cc:407] Connection header received
0619 18:45:44.465685 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.465690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.465759 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.465869 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.466368 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.467174 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.468663 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.469089 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.469242 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.470403 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.470409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.470412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:44.470433 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.470520 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:44.470544 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.470644 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.470820 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.470928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 18:45:44.475863 21116 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:44.483227 21116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:44.514740 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.496659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:41861 (local address 127.0.0.1:48614)
0619 18:45:44.499419 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.499459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.501044 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.502402 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.502413 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.502488 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.502982 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.502999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.507074 (+ 4075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.507083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.511312 (+ 4229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.511330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.513610 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.513617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.513869 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.513934 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.514109 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.514368 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":2293,"threads_started":1}
I20260619 18:45:44.514770 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.497117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.3.85.131:41861)
0619 18:45:44.498739 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.498753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.499564 (+ 811us) server_negotiation.cc:407] Connection header received
0619 18:45:44.501400 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.501413 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.501749 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.501952 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.503259 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.505242 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.511582 (+ 6340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.513392 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.513557 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.513901 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.514187 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.514367 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":818,"threads_started":1}
I20260619 18:45:44.519632 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.496925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51764)
0619 18:45:44.510062 (+ 13137us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.510093 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.510300 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.512377 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.512382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.512414 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.512990 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.513006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.515133 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.515139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.517217 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.517235 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.518685 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.518692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.518893 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.518931 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.519069 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.519220 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12915,"thread_start_us":13974,"threads_started":1}
I20260619 18:45:44.519735 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.497669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.3.85.129:33407)
0619 18:45:44.510951 (+ 13282us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.510960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.510996 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:45:44.511114 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.511123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.511840 (+ 717us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.512073 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.513191 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.514953 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.517426 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.518400 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.518611 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.518915 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.519129 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.519306 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13080,"thread_start_us":12934,"threads_started":1}
I20260619 18:45:44.519593 21499 tablet_service.cc:1467] Processing CreateTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 (DEFAULT_TABLE table=test-table [id=1ddee5b7ccb14ab18b432f7722d47082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.521363 21499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e251b3c4f7147cfb9ed068ae5bc7207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.523244 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.500656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:40495 (local address 127.0.0.1:54914)
0619 18:45:44.512113 (+ 11457us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.512140 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.512308 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.513880 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.513885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.513912 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.514347 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.514359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.517355 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.517360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.519430 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.519447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.521703 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.521712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.521894 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.521929 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.522006 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.522160 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11294,"thread_start_us":11157,"threads_started":1}
I20260619 18:45:44.523387 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.500089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.3.85.130:40495)
0619 18:45:44.512880 (+ 12791us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.512941 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.512976 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:45:44.513071 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.513077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.513405 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.513623 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.514593 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.517183 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.519695 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.521484 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.521624 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.522647 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.522753 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.522928 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12621,"thread_start_us":12500,"threads_started":1}
I20260619 18:45:44.523689 21238 tablet_service.cc:1467] Processing CreateTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 (DEFAULT_TABLE table=test-table [id=1ddee5b7ccb14ab18b432f7722d47082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.525329 21238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e251b3c4f7147cfb9ed068ae5bc7207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.527897 21369 tablet_service.cc:1467] Processing CreateTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 (DEFAULT_TABLE table=test-table [id=1ddee5b7ccb14ab18b432f7722d47082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:44.529556 21369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e251b3c4f7147cfb9ed068ae5bc7207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:44.537700 21583 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Bootstrap starting.
I20260619 18:45:44.543421 21584 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Bootstrap starting.
I20260619 18:45:44.544688 21583 tablet_bootstrap.cc:654] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.545051 21585 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Bootstrap starting.
I20260619 18:45:44.546293 21583 log.cc:824] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.547377 21584 tablet_bootstrap.cc:654] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.548533 21584 log.cc:824] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.548703 21585 tablet_bootstrap.cc:654] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:44.549970 21585 log.cc:824] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:44.551069 21583 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: No bootstrap required, opened a new log
I20260619 18:45:44.551502 21583 ts_tablet_manager.cc:1397] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260619 18:45:44.551833 21584 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: No bootstrap required, opened a new log
I20260619 18:45:44.552104 21584 ts_tablet_manager.cc:1397] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260619 18:45:44.553352 21585 tablet_bootstrap.cc:492] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: No bootstrap required, opened a new log
I20260619 18:45:44.553617 21585 ts_tablet_manager.cc:1397] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 18:45:44.559207 21584 raft_consensus.cc:348] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.559633 21584 raft_consensus.cc:374] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.559744 21584 raft_consensus.cc:729] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7057f5bdfd44e38f391dbbd050435e, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.560406 21584 consensus_queue.cc:260] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.562332 21584 ts_tablet_manager.cc:1428] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 18:45:44.563608 21583 raft_consensus.cc:348] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.564169 21583 raft_consensus.cc:374] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.564353 21583 raft_consensus.cc:729] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8543d621c0c4a22b0618bbc4ffd659b, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.564286 21585 raft_consensus.cc:348] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.564776 21585 raft_consensus.cc:374] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:44.564881 21585 raft_consensus.cc:729] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9b2d283b934cc9a4cea324cc1e00c9, State: Initialized, Role: FOLLOWER
I20260619 18:45:44.565106 21583 consensus_queue.cc:260] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.565495 21585 consensus_queue.cc:260] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.566666 21564 heartbeater.cc:503] Master 127.3.85.190:39883 was elected leader, sending a full tablet report...
I20260619 18:45:44.567029 21434 heartbeater.cc:503] Master 127.3.85.190:39883 was elected leader, sending a full tablet report...
I20260619 18:45:44.567210 21583 ts_tablet_manager.cc:1428] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260619 18:45:44.567623 21585 ts_tablet_manager.cc:1428] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260619 18:45:44.651378 21304 tablet.cc:2369] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:44.652388 21435 tablet.cc:2369] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:44.685338 21565 tablet.cc:2369] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:44.757586 21591 raft_consensus.cc:482] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:44.757912 21591 raft_consensus.cc:504] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.759743 21591 leader_election.cc:288] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8543d621c0c4a22b0618bbc4ffd659b (127.3.85.131:41861), 7a7057f5bdfd44e38f391dbbd050435e (127.3.85.129:33407)
I20260619 18:45:44.768450 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.760342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:40501 (local address 127.3.85.129:33407)
0619 18:45:44.760558 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.760566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.761038 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:45:44.761246 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.761249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.761320 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.761420 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.762082 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.763022 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.766009 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.767359 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.767563 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.767639 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.768039 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.768217 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:45:44.768536 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.760166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.3.85.130:40501)
0619 18:45:44.760919 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.760938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.761091 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.761434 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.761439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.761465 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.761914 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.761924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.763168 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.763173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.765739 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.765759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.767603 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.767610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.767853 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.767882 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.768010 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.768168 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":79,"thread_start_us":402,"threads_started":1}
I20260619 18:45:44.768693 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.760229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:34915 (local address 127.3.85.131:41861)
0619 18:45:44.760426 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.760434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.760940 (+ 506us) server_negotiation.cc:407] Connection header received
0619 18:45:44.761166 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.761170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.761252 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.761354 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:44.762166 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.763209 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.766594 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.767392 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.767596 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.767667 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.768284 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.768442 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:45:44.769412 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.759994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:41861 (local address 127.3.85.130:34915)
0619 18:45:44.760816 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:44.760832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:44.761019 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:44.761372 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:44.761376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:44.761400 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:44.761992 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.762002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.763340 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.763347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.765739 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:44.765759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.767801 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.767808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:44.768076 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.768112 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:45:44.768894 (+ 782us) client_negotiation.cc:240] Negotiation successful
0619 18:45:44.769051 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":124,"thread_start_us":450,"threads_started":1}
I20260619 18:45:44.769904 21258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e251b3c4f7147cfb9ed068ae5bc7207" candidate_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7057f5bdfd44e38f391dbbd050435e" is_pre_election: true
I20260619 18:45:44.770493 21519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e251b3c4f7147cfb9ed068ae5bc7207" candidate_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" is_pre_election: true
I20260619 18:45:44.770632 21258 raft_consensus.cc:2454] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9b2d283b934cc9a4cea324cc1e00c9 in term 0.
I20260619 18:45:44.771054 21519 raft_consensus.cc:2454] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9b2d283b934cc9a4cea324cc1e00c9 in term 0.
I20260619 18:45:44.771449 21322 leader_election.cc:302] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a7057f5bdfd44e38f391dbbd050435e, 9f9b2d283b934cc9a4cea324cc1e00c9; no voters:
I20260619 18:45:44.771946 21591 raft_consensus.cc:2781] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:44.772106 21591 raft_consensus.cc:482] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:44.772207 21591 raft_consensus.cc:3037] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.775547 21591 raft_consensus.cc:504] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.776335 21591 leader_election.cc:288] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [CANDIDATE]: Term 1 election: Requested vote from peers b8543d621c0c4a22b0618bbc4ffd659b (127.3.85.131:41861), 7a7057f5bdfd44e38f391dbbd050435e (127.3.85.129:33407)
I20260619 18:45:44.776965 21258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e251b3c4f7147cfb9ed068ae5bc7207" candidate_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7057f5bdfd44e38f391dbbd050435e"
I20260619 18:45:44.776984 21519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e251b3c4f7147cfb9ed068ae5bc7207" candidate_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8543d621c0c4a22b0618bbc4ffd659b"
I20260619 18:45:44.777220 21258 raft_consensus.cc:3037] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.777215 21519 raft_consensus.cc:3037] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:44.779734 21258 raft_consensus.cc:2454] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9b2d283b934cc9a4cea324cc1e00c9 in term 1.
I20260619 18:45:44.780287 21322 leader_election.cc:302] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a7057f5bdfd44e38f391dbbd050435e, 9f9b2d283b934cc9a4cea324cc1e00c9; no voters:
I20260619 18:45:44.780611 21591 raft_consensus.cc:2781] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:44.780628 21519 raft_consensus.cc:2454] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9b2d283b934cc9a4cea324cc1e00c9 in term 1.
I20260619 18:45:44.781198 21591 raft_consensus.cc:686] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9b2d283b934cc9a4cea324cc1e00c9, State: Running, Role: LEADER
I20260619 18:45:44.781724 21591 consensus_queue.cc:237] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } }
I20260619 18:45:44.787285 21115 catalog_manager.cc:5495] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9b2d283b934cc9a4cea324cc1e00c9 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9b2d283b934cc9a4cea324cc1e00c9" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 40495 } health_report { overall_health: HEALTHY } } }
I20260619 18:45:44.807994 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.802122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.3.85.130:40495)
0619 18:45:44.802325 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.802331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.802351 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:45:44.802644 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.802648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.802749 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.802916 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:44.803311 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.804015 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.805448 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.805871 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.806016 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.806640 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:44.807328 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:44.807509 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.807594 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.807680 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:45:44.818435 21258 raft_consensus.cc:1261] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 1 FOLLOWER]: Refusing update from remote peer 9f9b2d283b934cc9a4cea324cc1e00c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:44.818594 21519 raft_consensus.cc:1261] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 1 FOLLOWER]: Refusing update from remote peer 9f9b2d283b934cc9a4cea324cc1e00c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:44.819800 21594 consensus_queue.cc:1035] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7057f5bdfd44e38f391dbbd050435e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 33407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:44.820225 21591 consensus_queue.cc:1035] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8543d621c0c4a22b0618bbc4ffd659b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:44.835425 21599 mvcc.cc:205] Tried to move back new op lower bound from 7298640874764124160 to 7298640874630721536. Current Snapshot: MvccSnapshot[applied={T|T < 7298640874764124160}]
I20260619 18:45:44.852936 21116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:44.853258 21116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:44.857563 21116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c77b7e96d9449778e064d8b0d51f3f7: Sending DeleteTablet for 3 replicas of tablet 9e251b3c4f7147cfb9ed068ae5bc7207
I20260619 18:45:44.858935 21499 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='slave'} at 127.0.0.1:48614
I20260619 18:45:44.859033 21238 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='slave'} at 127.0.0.1:51764
I20260619 18:45:44.860245 21369 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e251b3c4f7147cfb9ed068ae5bc7207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:44 UTC) from {username='slave'} at 127.0.0.1:54914
I20260619 18:45:44.860325 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21176
I20260619 18:45:44.871060 21609 tablet_replica.cc:330] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: stopping tablet replica
I20260619 18:45:44.871706 21609 raft_consensus.cc:2229] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:44.872689 21609 raft_consensus.cc:2258] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.875442 21609 ts_tablet_manager.cc:1905] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.878134 21607 tablet_replica.cc:330] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: stopping tablet replica
I20260619 18:45:44.878476 21608 tablet_replica.cc:330] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: stopping tablet replica
I20260619 18:45:44.878729 21607 raft_consensus.cc:2229] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:44.879189 21608 raft_consensus.cc:2229] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:44.879309 21607 raft_consensus.cc:2258] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.879815 21608 raft_consensus.cc:2258] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:44.882311 21608 ts_tablet_manager.cc:1905] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 7a7057f5bdfd44e38f391dbbd050435e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.884541 21609 ts_tablet_manager.cc:1918] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:44.884661 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:44.862279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.3.85.129:33407)
0619 18:45:44.862560 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:44.862569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:44.872843 (+ 10274us) server_negotiation.cc:407] Connection header received
0619 18:45:44.873462 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:44.873469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:44.873604 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:44.873685 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:44.874467 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.875576 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.879814 (+ 4238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:44.880596 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:44.880769 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:44.882474 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:44.882488 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:44.882506 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:44.883312 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:44.883382 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:44.883418 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:44.883514 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:44.884084 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 18:45:44.884263 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:45:44.884747 21609 log.cc:1196] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/9e251b3c4f7147cfb9ed068ae5bc7207
I20260619 18:45:44.885389 21609 ts_tablet_manager.cc:1939] T 9e251b3c4f7147cfb9ed068ae5bc7207 P 9f9b2d283b934cc9a4cea324cc1e00c9: Deleting consensus metadata
I20260619 18:45:44.886833 21607 ts_tablet_manager.cc:1905] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:44.887460 21103 catalog_manager.cc:4841] TS 9f9b2d283b934cc9a4cea324cc1e00c9 (127.3.85.130:40495): tablet 9e251b3c4f7147cfb9ed068ae5bc7207 (table test-table [id=1ddee5b7ccb14ab18b432f7722d47082]) successfully deleted
I20260619 18:45:44.888955 21298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:44.895135 21607 ts_tablet_manager.cc:1918] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:44.895339 21607 log.cc:1196] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/9e251b3c4f7147cfb9ed068ae5bc7207
I20260619 18:45:44.895865 21607 ts_tablet_manager.cc:1939] T 9e251b3c4f7147cfb9ed068ae5bc7207 P b8543d621c0c4a22b0618bbc4ffd659b: Deleting consensus metadata
I20260619 18:45:44.898003 21103 catalog_manager.cc:4841] TS b8543d621c0c4a22b0618bbc4ffd659b (127.3.85.131:41861): tablet 9e251b3c4f7147cfb9ed068ae5bc7207 (table test-table [id=1ddee5b7ccb14ab18b432f7722d47082]) successfully deleted
I20260619 18:45:44.998654 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21176
W20260619 18:45:45.016757 21101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:33407 (error 108)
I20260619 18:45:45.016886 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21307
I20260619 18:45:45.019214 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.018642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51782)
0619 18:45:45.018885 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.019020 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 18:45:45.019538 21101 catalog_manager.cc:4584] TS 7a7057f5bdfd44e38f391dbbd050435e (127.3.85.129:33407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e251b3c4f7147cfb9ed068ae5bc7207: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
I20260619 18:45:45.026086 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.018425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.3.85.130:40495)
0619 18:45:45.018678 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:45.018685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:45.018705 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:45:45.018795 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:45.018801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:45.018904 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:45.019014 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:45.019534 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:45.020813 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:45.022140 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:45.022875 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:45.023054 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:45.024177 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:45.024190 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:45.024210 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:45.025123 (+ 913us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:45.025233 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:45.025261 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:45.025379 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:45.025549 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:45:45.025670 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 18:45:45.027057 21429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:45.073413 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.072886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51794)
0619 18:45:45.073126 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.073222 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 18:45:45.074608 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.074160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51810)
0619 18:45:45.074347 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.074421 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 18:45:45.135474 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.134935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51826)
0619 18:45:45.135182 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.135268 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 18:45:45.136636 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.136294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51832)
0619 18:45:45.136432 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.136504 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:45:45.147198 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21307
I20260619 18:45:45.167656 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21437
I20260619 18:45:45.174815 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.168371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.3.85.131:41861)
0619 18:45:45.168561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:45.168566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:45.168585 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:45:45.168707 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:45.168711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:45.168795 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:45.168892 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:45.169348 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:45.170052 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:45.171475 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:45.171880 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:45.171980 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:45.173156 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:45.173165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:45.173181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:45.173801 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:45.173869 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:45.173892 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:45.173993 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:45.174147 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:45:45.174435 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:45:45.175673 21559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:45.217294 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.216766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51846)
0619 18:45:45.217009 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.217100 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 18:45:45.218592 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.218043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51858)
0619 18:45:45.218260 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.218380 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 18:45:45.266928 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21437
I20260619 18:45:45.284844 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21086
I20260619 18:45:45.286348 21147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:45.399777 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21086
I20260619 18:45:45.399917 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:45.399184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:33407 (local address 127.0.0.1:51866)
0619 18:45:45.399584 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:45.399702 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:33407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
2026-06-19T18:45:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3177 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:45 dist-test-slave-09fp krb5kdc[21618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:45 dist-test-slave-09fp krb5kdc[21618](info): set up 2 sockets
Jun 19 18:45:45 dist-test-slave-09fp krb5kdc[21618](info): commencing operation
krb5kdc: starting...
W20260619 18:45:46.526641 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:45:46 dist-test-slave-09fp krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:45:46.793710 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39197
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:47.153231 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:47.153621 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:47.153798 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:47.161506 21631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:47.161701 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:47.161823 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:47.161924 21631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:47.162014 21631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:47.172622 21631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:47.174165 21631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:47.175693 21631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:47.188870 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:47.188777 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:47.188802 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:47.189844 21631 server_base.cc:1034] running on GCE node
I20260619 18:45:47.190891 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:47.192516 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:47.193806 21631 hybrid_clock.cc:648] HybridClock initialized: now 1781894747193763 us; error 61 us; skew 500 ppm
I20260619 18:45:47.194231 21631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:47 dist-test-slave-09fp krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:47.197834 21631 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:45:47.199733 21631 webserver.cc:460] Webserver started at http://127.3.85.190:42109/ using document root <none> and password file <none>
I20260619 18:45:47.200537 21631 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:47.200716 21631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:47.201206 21631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:47.204728 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "83940e9dace744f4bc319f533b1364e2"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "4a612a7690c09b5ec2fa9befe205d7b6"
server_key_iv: "bc1ece379807d57a29556feaf298076c"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.205667 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "83940e9dace744f4bc319f533b1364e2"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "4a612a7690c09b5ec2fa9befe205d7b6"
server_key_iv: "bc1ece379807d57a29556feaf298076c"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.211802 21631 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:45:47.216084 21646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:47.217672 21631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:47.217928 21631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "83940e9dace744f4bc319f533b1364e2"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "4a612a7690c09b5ec2fa9befe205d7b6"
server_key_iv: "bc1ece379807d57a29556feaf298076c"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.218189 21631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:47.242282 21631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:47.247156 21631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:47.247514 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:47.267771 21631 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39197
I20260619 18:45:47.267808 21697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39197 every 8 connection(s)
I20260619 18:45:47.269245 21631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:47.271157 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21631
I20260619 18:45:47.271361 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:47.271956 3414 external_mini_cluster.cc:1428] Setting key 604b005cbaeab174e8d0b1c5c82ffd9c
I20260619 18:45:47.274205 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:45:47 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:47.288975 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: Bootstrap starting.
I20260619 18:45:47.293198 21698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:47.294405 21698 log.cc:824] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:47.298523 21698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: No bootstrap required, opened a new log
I20260619 18:45:47.298779 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:47.273957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.3.85.190:39197)
0619 18:45:47.274833 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:47.274844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:47.274888 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:45:47.276620 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:47.276646 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:47.277097 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:47.277487 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:47.279506 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.281010 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.283095 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.283858 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.284046 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:47.289206 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:47.289226 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:47.289245 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:47.289293 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:47.293223 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:47.294147 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:47.294165 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:47.294173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:47.294367 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:47.294920 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:47.294932 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:47.294935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:47.295854 (+ 919us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:47.296179 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:47.296735 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 18:45:47.297186 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":427,"threads_started":1}
I20260619 18:45:47.307601 21698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } }
I20260619 18:45:47.308065 21698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:47.308189 21698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83940e9dace744f4bc319f533b1364e2, State: Initialized, Role: FOLLOWER
I20260619 18:45:47.308809 21698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } }
I20260619 18:45:47.309640 21698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:47.309844 21698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:47.310057 21698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:47.312536 21698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } }
I20260619 18:45:47.313217 21698 leader_election.cc:302] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83940e9dace744f4bc319f533b1364e2; no voters:
I20260619 18:45:47.313897 21698 leader_election.cc:288] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:47.314082 21703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:47.314934 21703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [term 1 LEADER]: Becoming Leader. State: Replica: 83940e9dace744f4bc319f533b1364e2, State: Running, Role: LEADER
I20260619 18:45:47.315634 21703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } }
I20260619 18:45:47.316381 21698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:47.318408 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83940e9dace744f4bc319f533b1364e2. Latest consensus state: current_term: 1 leader_uuid: "83940e9dace744f4bc319f533b1364e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } } }
I20260619 18:45:47.318897 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:47.318811 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83940e9dace744f4bc319f533b1364e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83940e9dace744f4bc319f533b1364e2" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39197 } } }
I20260619 18:45:47.319099 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:47.321476 21709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:47.338142 21709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:47.348690 21709 catalog_manager.cc:1347] Generated new cluster ID: 1377db8b013846a5b4eb1f4aa441b8f0
I20260619 18:45:47.348843 21709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:47.374226 21709 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:47.375727 21709 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:47.383833 21709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: Generated new TSK 0
I20260619 18:45:47.384739 21709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:45:47.458851 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:47.816061 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:47.816524 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:47.816695 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:47.826490 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:47.826697 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:47.826900 21724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:47.837664 21724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:47.839206 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:47.840590 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:47.855355 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:47.855404 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:47.855587 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:47.856328 21724 server_base.cc:1034] running on GCE node
I20260619 18:45:47.857015 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:47.858047 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:47.859324 21724 hybrid_clock.cc:648] HybridClock initialized: now 1781894747859265 us; error 52 us; skew 500 ppm
I20260619 18:45:47.859728 21724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:47 dist-test-slave-09fp krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:47.866426 21724 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:45:47.868340 21724 webserver.cc:460] Webserver started at http://127.3.85.129:39455/ using document root <none> and password file <none>
I20260619 18:45:47.869130 21724 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:47.869297 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:47.869638 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:47.873327 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "27dd03d0fc2342859859e71d89510d3e"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "6ef2b76dae594766367a69f83ec92943"
server_key_iv: "f52e07569b399b9ae3473058b617615a"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.874257 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "27dd03d0fc2342859859e71d89510d3e"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "6ef2b76dae594766367a69f83ec92943"
server_key_iv: "f52e07569b399b9ae3473058b617615a"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.880131 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:45:47.884058 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:47.885564 21724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:45:47.885762 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "27dd03d0fc2342859859e71d89510d3e"
format_stamp: "Formatted at 2026-06-19 18:45:47 on dist-test-slave-09fp"
server_key: "6ef2b76dae594766367a69f83ec92943"
server_key_iv: "f52e07569b399b9ae3473058b617615a"
server_key_version: "encryptionkey@0"
I20260619 18:45:47.886003 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:47.917088 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:47.922333 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:47.922653 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:47.923792 21724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:47.925293 21724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:47.925375 21724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:47.925474 21724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:47.925531 21724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:47.961090 21724 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:37153
I20260619 18:45:47.961143 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:37153 every 8 connection(s)
I20260619 18:45:47.962550 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:47.969511 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21724
I20260619 18:45:47.969682 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:47.970175 3414 external_mini_cluster.cc:1428] Setting key 44d89d4784736d4c1c5043d214e30369
Jun 19 18:45:47 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894747, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:47.985767 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:47.965285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:53113 (local address 127.3.85.190:39197)
0619 18:45:47.965794 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:47.965801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:47.966952 (+ 1151us) server_negotiation.cc:407] Connection header received
0619 18:45:47.968846 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:47.968851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:47.968923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:47.969029 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:47.970997 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.971883 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.973678 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.974494 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.974658 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:47.978491 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:47.978501 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:47.978509 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:47.978563 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:47.981647 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:47.982918 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:47.982923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:47.982925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:47.983067 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:47.983731 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:47.983738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:47.983741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:47.984117 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:47.984242 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:47.985305 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0619 18:45:47.985502 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":291,"threads_started":1}
I20260619 18:45:47.987584 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:47.965928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39197 (local address 127.3.85.129:53113)
0619 18:45:47.966779 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:47.966819 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:47.968559 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:47.969304 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:47.969313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:47.969979 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:47.970753 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:47.970781 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.972108 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.972114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:47.973495 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:47.973510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:47.974740 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:47.974745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:47.974976 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:47.975764 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:47.975790 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:47.978157 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:47.981879 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:47.981896 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:47.981919 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:47.982764 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:47.983204 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:47.983213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:47.983219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:47.983561 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:47.984275 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:47.984289 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:47.984858 (+ 569us) client_negotiation.cc:769] Sending connection context
0619 18:45:47.985197 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 18:45:47.985863 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":332,"threads_started":1}
I20260619 18:45:47.989630 21852 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39197
I20260619 18:45:47.990020 21852 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:47.990981 21852 heartbeater.cc:511] Master 127.3.85.190:39197 requested a full tablet report, sending...
I20260619 18:45:47.993461 21663 ts_manager.cc:194] Registered new tserver with Master: 27dd03d0fc2342859859e71d89510d3e (127.3.85.129:37153)
I20260619 18:45:47.995406 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:53113
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:45:48.030417 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:48.376065 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:48.376439 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:48.376627 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:48.384447 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:48.384629 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:48.384816 21858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:48.395572 21858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:48.396982 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:48.398399 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:48.412967 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:48.412917 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:48.413012 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:48.414106 21858 server_base.cc:1034] running on GCE node
I20260619 18:45:48.414917 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:48.416049 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:48.417315 21858 hybrid_clock.cc:648] HybridClock initialized: now 1781894748417261 us; error 39 us; skew 500 ppm
I20260619 18:45:48.417886 21858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:48 dist-test-slave-09fp krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:48.421766 21858 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:45:48.423956 21858 webserver.cc:460] Webserver started at http://127.3.85.130:45533/ using document root <none> and password file <none>
I20260619 18:45:48.424778 21858 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:48.424955 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:48.425297 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:48.428613 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "8ce55754e487428681bd57e2693f9cf7"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "ba251689dc25d6f8ec744b1fb6ae2c33"
server_key_iv: "b9cae28e1c3ec0f7bade6466155a6971"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.429576 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce55754e487428681bd57e2693f9cf7"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "ba251689dc25d6f8ec744b1fb6ae2c33"
server_key_iv: "b9cae28e1c3ec0f7bade6466155a6971"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.435770 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 18:45:48.440049 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:48.441699 21858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:45:48.441939 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "8ce55754e487428681bd57e2693f9cf7"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "ba251689dc25d6f8ec744b1fb6ae2c33"
server_key_iv: "b9cae28e1c3ec0f7bade6466155a6971"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.442202 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:48.456825 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:48.461521 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:48.461864 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:48.462929 21858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:48.464450 21858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:48.464560 21858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:48.464720 21858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:48.464800 21858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:48.500319 21858 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:43597
I20260619 18:45:48.500377 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:43597 every 8 connection(s)
I20260619 18:45:48.501770 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:48.507453 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21858
I20260619 18:45:48.507650 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:48.508121 3414 external_mini_cluster.cc:1428] Setting key 900f3ca3f60ffcd2c65e61359c840619
Jun 19 18:45:48 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:48.525813 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:48.504744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:35729 (local address 127.3.85.190:39197)
0619 18:45:48.505245 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:48.505251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:48.506439 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 18:45:48.508655 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:48.508664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:48.508753 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:48.508853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:48.510745 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:48.511956 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:48.513769 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:48.514577 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:48.514737 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:48.518873 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:48.518879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:48.518882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:48.518922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:48.521871 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:48.523106 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:48.523111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:48.523113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:48.523261 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:48.523875 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:48.523879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:48.523881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:48.524163 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:48.524283 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:48.525311 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0619 18:45:48.525537 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":247,"threads_started":1}
I20260619 18:45:48.527489 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:48.505364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39197 (local address 127.3.85.130:35729)
0619 18:45:48.506238 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:48.506292 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:48.508321 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:48.509141 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:48.509150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:48.509706 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:48.510555 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:48.510578 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:48.512173 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:48.512178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:48.513574 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:48.513590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:48.514841 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:48.514847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:48.515091 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:48.515944 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:48.515969 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:48.518579 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:48.522067 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:48.522082 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:48.522107 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:48.522939 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:48.523395 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:48.523403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:48.523409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:48.523696 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:48.524308 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:48.524320 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:48.524858 (+ 538us) client_negotiation.cc:769] Sending connection context
0619 18:45:48.525183 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 18:45:48.525845 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":334,"threads_started":1}
I20260619 18:45:48.529141 21986 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39197
I20260619 18:45:48.529487 21986 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:48.530275 21986 heartbeater.cc:511] Master 127.3.85.190:39197 requested a full tablet report, sending...
I20260619 18:45:48.532112 21663 ts_manager.cc:194] Registered new tserver with Master: 8ce55754e487428681bd57e2693f9cf7 (127.3.85.130:43597)
I20260619 18:45:48.533548 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:35729
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:45:48.570550 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:48.922031 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:48.922343 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:48.922564 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:48.930335 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:48.930544 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:48.930759 21992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:48.941103 21992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:48.942582 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:48.944013 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:48.958441 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:48.958527 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:48.958810 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:48.959470 21992 server_base.cc:1034] running on GCE node
I20260619 18:45:48.960191 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:48.961369 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:48.962656 21992 hybrid_clock.cc:648] HybridClock initialized: now 1781894748962603 us; error 54 us; skew 500 ppm
I20260619 18:45:48.963008 21992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:48 dist-test-slave-09fp krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:48.966773 21992 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:45:48.968353 21992 webserver.cc:460] Webserver started at http://127.3.85.131:44943/ using document root <none> and password file <none>
I20260619 18:45:48.969064 21992 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:48.969170 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:48.969448 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:48.972668 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "709a9a18cbaf4436bd7dbc8b77686a1b"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "614dc42f214d7eee5fc84b1a4430df0b"
server_key_iv: "e765fc4f39cb9db038b98510cc4618c7"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.973520 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "709a9a18cbaf4436bd7dbc8b77686a1b"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "614dc42f214d7eee5fc84b1a4430df0b"
server_key_iv: "e765fc4f39cb9db038b98510cc4618c7"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.979547 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:45:48.983570 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:48.985081 21992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:45:48.985303 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "709a9a18cbaf4436bd7dbc8b77686a1b"
format_stamp: "Formatted at 2026-06-19 18:45:48 on dist-test-slave-09fp"
server_key: "614dc42f214d7eee5fc84b1a4430df0b"
server_key_iv: "e765fc4f39cb9db038b98510cc4618c7"
server_key_version: "encryptionkey@0"
I20260619 18:45:48.985539 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:48.999161 21852 heartbeater.cc:503] Master 127.3.85.190:39197 was elected leader, sending a full tablet report...
I20260619 18:45:49.008315 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:49.012956 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:49.013293 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:49.014407 21992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:49.016031 21992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:49.016119 21992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:49.016311 21992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:49.016363 21992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:49.052629 21992 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:39967
I20260619 18:45:49.052750 22119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:39967 every 8 connection(s)
I20260619 18:45:49.054226 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:49.057070 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 21992
I20260619 18:45:49.057274 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:49.057857 3414 external_mini_cluster.cc:1428] Setting key 4b67ee050b6754c475e261306e1af521
Jun 19 18:45:49 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894748, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:49.077077 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.057424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:59985 (local address 127.3.85.190:39197)
0619 18:45:49.058000 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.058008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.059231 (+ 1223us) server_negotiation.cc:407] Connection header received
0619 18:45:49.061689 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.061693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.061776 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.061870 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:49.063604 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.064382 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.065985 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.066401 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.066600 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.070241 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:49.070247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:49.070251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:49.070290 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:49.073917 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.074855 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.074860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.074862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.075002 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.075411 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.075414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.075416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.075719 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:49.075842 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.076599 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.076787 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":265,"threads_started":1}
I20260619 18:45:49.078171 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.058035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39197 (local address 127.3.85.131:59985)
0619 18:45:49.059060 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.059101 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.061403 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.062116 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.062125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.062788 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:49.063421 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.063449 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.064525 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.064529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.065828 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.065841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.066736 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.066742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.066972 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.067651 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:49.067680 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:49.069987 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:49.074088 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:49.074097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:49.074114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:49.074731 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:49.075106 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:49.075109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:49.075111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:49.075303 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:49.075849 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:49.075856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:49.076232 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.076490 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.076916 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":435,"threads_started":1}
I20260619 18:45:49.079898 22120 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39197
I20260619 18:45:49.080251 22120 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:49.081033 22120 heartbeater.cc:511] Master 127.3.85.190:39197 requested a full tablet report, sending...
I20260619 18:45:49.082705 21663 ts_manager.cc:194] Registered new tserver with Master: 709a9a18cbaf4436bd7dbc8b77686a1b (127.3.85.131:39967)
I20260619 18:45:49.083559 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:59985
I20260619 18:45:49.086367 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:49.104687 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.090790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.3.85.190:39197)
0619 18:45:49.090959 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.090963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.091552 (+ 589us) server_negotiation.cc:407] Connection header received
0619 18:45:49.091705 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.091709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.091789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.091892 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:49.093434 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.094317 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.095681 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.096098 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.096203 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.099119 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:49.099124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:49.099127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:49.099164 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:49.101744 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.102822 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.102826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.102828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.102963 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.103394 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.103397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.103399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.103731 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:49.103947 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.104293 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.104444 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:45:49.109052 21663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:49.116196 21663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:49.156426 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.130055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.3.85.131:39967)
0619 18:45:49.146431 (+ 16376us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.146460 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.146502 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:45:49.146627 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.146639 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.147149 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.147388 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:49.148811 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.150453 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.153302 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.155085 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.155227 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.155485 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.155821 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.156016 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16159,"thread_start_us":15965,"threads_started":1}
I20260619 18:45:49.156558 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.129774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:39967 (local address 127.0.0.1:40878)
0619 18:45:49.144146 (+ 14372us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.144222 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.144498 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.147843 (+ 3345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.147855 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.147902 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:49.148506 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.148523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.150600 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.150605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.153098 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.153114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.155299 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.155306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.155548 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.155603 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.155758 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.156040 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13951,"mutex_wait_us":5,"thread_start_us":14972,"threads_started":1}
I20260619 18:45:49.156842 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.133114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.3.85.129:37153)
0619 18:45:49.148252 (+ 15138us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.148261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.148922 (+ 661us) server_negotiation.cc:407] Connection header received
0619 18:45:49.149250 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.149259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.149500 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.149665 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:49.150509 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.151841 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.154243 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.155368 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.155506 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.155728 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.156293 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.156491 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14970,"thread_start_us":14795,"threads_started":1}
I20260619 18:45:49.157226 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.129862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:43597 (local address 127.0.0.1:56766)
0619 18:45:49.147415 (+ 17553us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.147446 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.147621 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.148845 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.148853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.148885 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:49.149321 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.149335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.151762 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.151767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.154264 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.154281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.156289 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.156295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.156520 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.156562 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.156641 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.156776 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17381,"mutex_wait_us":22,"spinlock_wait_cycles":11008,"thread_start_us":18628,"threads_started":1}
I20260619 18:45:49.157668 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.132805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:37153 (local address 127.0.0.1:41030)
0619 18:45:49.148803 (+ 15998us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.148826 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.149045 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.149942 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.149948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.149979 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:49.150305 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.150320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.152011 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.152015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.154091 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.154107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.155869 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.155875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.156076 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.156106 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.157058 (+ 952us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.157231 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15857,"mutex_wait_us":56,"thread_start_us":14272,"threads_started":1}
I20260619 18:45:49.159334 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.130177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.3.85.130:43597)
0619 18:45:49.145535 (+ 15358us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.145548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.148214 (+ 2666us) server_negotiation.cc:407] Connection header received
0619 18:45:49.148340 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.148352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.148623 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.149573 (+ 950us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:49.149863 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.151587 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.154555 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.156023 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.157784 (+ 1761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.158858 (+ 1074us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.158942 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.159079 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15111,"thread_start_us":15485,"threads_started":1}
I20260619 18:45:49.161541 21787 tablet_service.cc:1467] Processing CreateTablet for tablet 823b2b0022624ad3b089d8a5721fa417 (DEFAULT_TABLE table=test-table [id=93087f43cfe64a21ad6babb1101e2243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:49.161541 22055 tablet_service.cc:1467] Processing CreateTablet for tablet 823b2b0022624ad3b089d8a5721fa417 (DEFAULT_TABLE table=test-table [id=93087f43cfe64a21ad6babb1101e2243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:49.162099 21921 tablet_service.cc:1467] Processing CreateTablet for tablet 823b2b0022624ad3b089d8a5721fa417 (DEFAULT_TABLE table=test-table [id=93087f43cfe64a21ad6babb1101e2243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:49.163170 22055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823b2b0022624ad3b089d8a5721fa417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:49.163419 21921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823b2b0022624ad3b089d8a5721fa417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:49.163746 21787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823b2b0022624ad3b089d8a5721fa417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:49.175284 22139 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Bootstrap starting.
I20260619 18:45:49.179032 22140 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Bootstrap starting.
I20260619 18:45:49.180212 22139 tablet_bootstrap.cc:654] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:49.181051 22141 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Bootstrap starting.
I20260619 18:45:49.181780 22139 log.cc:824] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:49.182955 22140 tablet_bootstrap.cc:654] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:49.184213 22140 log.cc:824] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:49.184492 22141 tablet_bootstrap.cc:654] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:49.185420 22139 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: No bootstrap required, opened a new log
I20260619 18:45:49.185756 22139 ts_tablet_manager.cc:1397] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 18:45:49.185781 22141 log.cc:824] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:49.187487 22140 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: No bootstrap required, opened a new log
I20260619 18:45:49.187865 22140 ts_tablet_manager.cc:1397] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 18:45:49.189785 22141 tablet_bootstrap.cc:492] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: No bootstrap required, opened a new log
I20260619 18:45:49.190130 22141 ts_tablet_manager.cc:1397] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 18:45:49.197762 22141 raft_consensus.cc:348] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.198315 22141 raft_consensus.cc:374] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:49.198482 22141 raft_consensus.cc:729] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce55754e487428681bd57e2693f9cf7, State: Initialized, Role: FOLLOWER
I20260619 18:45:49.198990 22139 raft_consensus.cc:348] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.199064 22140 raft_consensus.cc:348] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.199452 22139 raft_consensus.cc:374] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:49.199483 22140 raft_consensus.cc:374] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:49.199307 22141 consensus_queue.cc:260] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.199553 22139 raft_consensus.cc:729] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709a9a18cbaf4436bd7dbc8b77686a1b, State: Initialized, Role: FOLLOWER
I20260619 18:45:49.199587 22140 raft_consensus.cc:729] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dd03d0fc2342859859e71d89510d3e, State: Initialized, Role: FOLLOWER
I20260619 18:45:49.200268 22139 consensus_queue.cc:260] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.200354 22140 consensus_queue.cc:260] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.200721 21986 heartbeater.cc:503] Master 127.3.85.190:39197 was elected leader, sending a full tablet report...
I20260619 18:45:49.201653 22141 ts_tablet_manager.cc:1428] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 18:45:49.203469 22140 ts_tablet_manager.cc:1428] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 18:45:49.205199 22120 heartbeater.cc:503] Master 127.3.85.190:39197 was elected leader, sending a full tablet report...
I20260619 18:45:49.205914 22139 ts_tablet_manager.cc:1428] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
W20260619 18:45:49.215152 21853 tablet.cc:2369] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:49.254079 21987 tablet.cc:2369] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:49.306190 22121 tablet.cc:2369] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:49.326884 22145 raft_consensus.cc:482] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:49.327241 22145 raft_consensus.cc:504] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.329099 22145 leader_election.cc:288] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709a9a18cbaf4436bd7dbc8b77686a1b (127.3.85.131:39967), 27dd03d0fc2342859859e71d89510d3e (127.3.85.129:37153)
I20260619 18:45:49.336531 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.329799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:33461 (local address 127.3.85.129:37153)
0619 18:45:49.330031 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.330039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.330829 (+ 790us) server_negotiation.cc:407] Connection header received
0619 18:45:49.330999 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.331003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.331104 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.331205 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:49.331862 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.332619 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.334565 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.335418 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.335631 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.335708 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.336093 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.336240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:45:49.336557 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.329440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:51137 (local address 127.3.85.131:39967)
0619 18:45:49.329742 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.329752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.330679 (+ 927us) server_negotiation.cc:407] Connection header received
0619 18:45:49.330849 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.330855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.330955 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.331045 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:49.331734 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.332525 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.334580 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.335390 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.335576 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.335673 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.336162 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.336313 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:45:49.336668 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.329564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:37153 (local address 127.3.85.130:33461)
0619 18:45:49.330670 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.330690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.330861 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.331243 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.331246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.331278 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:49.331694 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.331707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.332763 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.332767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.334357 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.334371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.335696 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.335705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.335860 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.335909 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.335995 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.336155 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":48,"thread_start_us":687,"threads_started":1}
I20260619 18:45:49.336701 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.329297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:39967 (local address 127.3.85.130:51137)
0619 18:45:49.330541 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:49.330562 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:49.330703 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:49.331089 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:49.331094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:49.331119 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:49.331568 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.331581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.332664 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.332673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.334350 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:49.334365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.335655 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.335663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:49.335847 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.335950 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 18:45:49.336101 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:45:49.336239 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":88,"thread_start_us":814,"threads_started":1}
I20260619 18:45:49.337855 22075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823b2b0022624ad3b089d8a5721fa417" candidate_uuid: "8ce55754e487428681bd57e2693f9cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" is_pre_election: true
I20260619 18:45:49.338045 21807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823b2b0022624ad3b089d8a5721fa417" candidate_uuid: "8ce55754e487428681bd57e2693f9cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd03d0fc2342859859e71d89510d3e" is_pre_election: true
I20260619 18:45:49.338548 22075 raft_consensus.cc:2454] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce55754e487428681bd57e2693f9cf7 in term 0.
I20260619 18:45:49.338587 21807 raft_consensus.cc:2454] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce55754e487428681bd57e2693f9cf7 in term 0.
I20260619 18:45:49.339442 21876 leader_election.cc:302] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dd03d0fc2342859859e71d89510d3e, 8ce55754e487428681bd57e2693f9cf7; no voters:
I20260619 18:45:49.340056 22145 raft_consensus.cc:2781] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:49.340236 22145 raft_consensus.cc:482] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:49.340377 22145 raft_consensus.cc:3037] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:49.342810 22145 raft_consensus.cc:504] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.343564 22145 leader_election.cc:288] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [CANDIDATE]: Term 1 election: Requested vote from peers 709a9a18cbaf4436bd7dbc8b77686a1b (127.3.85.131:39967), 27dd03d0fc2342859859e71d89510d3e (127.3.85.129:37153)
I20260619 18:45:49.344111 22075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823b2b0022624ad3b089d8a5721fa417" candidate_uuid: "8ce55754e487428681bd57e2693f9cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b"
I20260619 18:45:49.344161 21807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823b2b0022624ad3b089d8a5721fa417" candidate_uuid: "8ce55754e487428681bd57e2693f9cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27dd03d0fc2342859859e71d89510d3e"
I20260619 18:45:49.344336 22075 raft_consensus.cc:3037] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:49.344437 21807 raft_consensus.cc:3037] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:49.347013 21807 raft_consensus.cc:2454] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce55754e487428681bd57e2693f9cf7 in term 1.
I20260619 18:45:49.347019 22075 raft_consensus.cc:2454] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce55754e487428681bd57e2693f9cf7 in term 1.
I20260619 18:45:49.347599 21876 leader_election.cc:302] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dd03d0fc2342859859e71d89510d3e, 8ce55754e487428681bd57e2693f9cf7; no voters:
I20260619 18:45:49.347970 22145 raft_consensus.cc:2781] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:49.348428 22145 raft_consensus.cc:686] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce55754e487428681bd57e2693f9cf7, State: Running, Role: LEADER
I20260619 18:45:49.348938 22145 consensus_queue.cc:237] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } }
I20260619 18:45:49.354079 21663 catalog_manager.cc:5495] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce55754e487428681bd57e2693f9cf7 (127.3.85.130). New cstate: current_term: 1 leader_uuid: "8ce55754e487428681bd57e2693f9cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce55754e487428681bd57e2693f9cf7" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 43597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:49.400075 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.394685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.3.85.130:43597)
0619 18:45:49.394932 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.394939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.394967 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:45:49.395056 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.395060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.395138 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.395277 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:49.395649 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.396362 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.397762 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.398216 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.398327 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.398894 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:49.399415 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:49.399551 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.399690 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.399767 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 18:45:49.409250 21807 raft_consensus.cc:1261] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 1 FOLLOWER]: Refusing update from remote peer 8ce55754e487428681bd57e2693f9cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:49.409262 22075 raft_consensus.cc:1261] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 1 FOLLOWER]: Refusing update from remote peer 8ce55754e487428681bd57e2693f9cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:49.410422 22150 consensus_queue.cc:1035] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27dd03d0fc2342859859e71d89510d3e" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:49.410868 22145 consensus_queue.cc:1035] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "709a9a18cbaf4436bd7dbc8b77686a1b" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:49.425315 22155 mvcc.cc:205] Tried to move back new op lower bound from 7298640893568131072 to 7298640893337436160. Current Snapshot: MvccSnapshot[applied={T|T < 7298640893568131072}]
I20260619 18:45:49.438503 21663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:49.438799 21663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:49.442649 21663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83940e9dace744f4bc319f533b1364e2: Sending DeleteTablet for 3 replicas of tablet 823b2b0022624ad3b089d8a5721fa417
I20260619 18:45:49.444079 21787 tablet_service.cc:1514] Processing DeleteTablet for tablet 823b2b0022624ad3b089d8a5721fa417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:49 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:41030
I20260619 18:45:49.444057 21921 tablet_service.cc:1514] Processing DeleteTablet for tablet 823b2b0022624ad3b089d8a5721fa417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:49 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:56766
I20260619 18:45:49.444100 22055 tablet_service.cc:1514] Processing DeleteTablet for tablet 823b2b0022624ad3b089d8a5721fa417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:49 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:40878
I20260619 18:45:49.446062 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21724
I20260619 18:45:49.470378 22164 tablet_replica.cc:330] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: stopping tablet replica
I20260619 18:45:49.471238 22164 raft_consensus.cc:2229] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:49.472047 22164 raft_consensus.cc:2258] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:49.475984 22163 tablet_replica.cc:330] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: stopping tablet replica
I20260619 18:45:49.476902 22165 tablet_replica.cc:330] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: stopping tablet replica
I20260619 18:45:49.477052 22163 raft_consensus.cc:2229] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:49.477463 22165 raft_consensus.cc:2229] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:49.477665 22164 ts_tablet_manager.cc:1905] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:49.477849 22163 raft_consensus.cc:2258] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:49.478121 22165 raft_consensus.cc:2258] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:49.480712 22165 ts_tablet_manager.cc:1905] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:45:49 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:45:49.482903 22163 ts_tablet_manager.cc:1905] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:49.485735 22164 ts_tablet_manager.cc:1918] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:49.485993 22164 log.cc:1196] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/823b2b0022624ad3b089d8a5721fa417
I20260619 18:45:49.486662 22164 ts_tablet_manager.cc:1939] T 823b2b0022624ad3b089d8a5721fa417 P 27dd03d0fc2342859859e71d89510d3e: Deleting consensus metadata
I20260619 18:45:49.488895 21649 catalog_manager.cc:4841] TS 27dd03d0fc2342859859e71d89510d3e (127.3.85.129:37153): tablet 823b2b0022624ad3b089d8a5721fa417 (table test-table [id=93087f43cfe64a21ad6babb1101e2243]) successfully deleted
I20260619 18:45:49.491330 22163 ts_tablet_manager.cc:1918] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:49.491686 22163 log.cc:1196] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/823b2b0022624ad3b089d8a5721fa417
I20260619 18:45:49.492425 22163 ts_tablet_manager.cc:1939] T 823b2b0022624ad3b089d8a5721fa417 P 8ce55754e487428681bd57e2693f9cf7: Deleting consensus metadata
I20260619 18:45:49.493562 22165 ts_tablet_manager.cc:1918] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:49.493780 22165 log.cc:1196] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/823b2b0022624ad3b089d8a5721fa417
I20260619 18:45:49.494343 22165 ts_tablet_manager.cc:1939] T 823b2b0022624ad3b089d8a5721fa417 P 709a9a18cbaf4436bd7dbc8b77686a1b: Deleting consensus metadata
I20260619 18:45:49.494695 21650 catalog_manager.cc:4841] TS 8ce55754e487428681bd57e2693f9cf7 (127.3.85.130:43597): tablet 823b2b0022624ad3b089d8a5721fa417 (table test-table [id=93087f43cfe64a21ad6babb1101e2243]) successfully deleted
I20260619 18:45:49.495759 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.449139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.3.85.129:37153)
0619 18:45:49.449391 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.449401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.461085 (+ 11684us) server_negotiation.cc:407] Connection header received
0619 18:45:49.461714 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.461726 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.461853 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.461980 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:49.464567 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.466260 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.472340 (+ 6080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.473366 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.473512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.487061 (+ 13549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:49.487077 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:49.487099 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:49.487156 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:49.491505 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.492631 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.492643 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.492650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.492840 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.493373 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.493380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.493383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.493894 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:49.494212 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.495260 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.495465 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:45:49.496313 21648 catalog_manager.cc:4841] TS 709a9a18cbaf4436bd7dbc8b77686a1b (127.3.85.131:39967): tablet 823b2b0022624ad3b089d8a5721fa417 (table test-table [id=93087f43cfe64a21ad6babb1101e2243]) successfully deleted
I20260619 18:45:49.497319 21847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:49.616685 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21724
I20260619 18:45:49.635434 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21858
Jun 19 18:45:49 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:45:49.652717 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.636258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.3.85.130:43597)
0619 18:45:49.636435 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.636441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.636532 (+ 91us) server_negotiation.cc:407] Connection header received
0619 18:45:49.636745 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.636751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.636848 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.637019 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:49.638332 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.639248 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.640633 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.641111 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.641265 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.645970 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:49.645990 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:49.646016 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:49.646068 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:49.649539 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.650387 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.650395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.650397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.650625 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.651017 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.651024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.651027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.651430 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:49.651768 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.652083 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.652324 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:45:49.653545 21981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:49.770620 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21858
I20260619 18:45:49.789117 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21992
Jun 19 18:45:49 dist-test-slave-09fp krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:45:49.805320 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:49.789865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.3.85.131:39967)
0619 18:45:49.790061 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:49.790067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:49.790186 (+ 119us) server_negotiation.cc:407] Connection header received
0619 18:45:49.790391 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:49.790401 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:49.790531 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:49.790651 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:49.792043 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.792755 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.794030 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:49.794443 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:49.794659 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:49.799491 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:49.799510 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:49.799527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:49.799571 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:49.802208 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.803132 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.803140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.803143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.803307 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:49.803711 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:49.803716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:49.803719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:49.804098 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:49.804412 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:49.804729 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 18:45:49.804973 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:45:49.806063 22115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:49.898855 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21992
I20260619 18:45:49.916054 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 21631
I20260619 18:45:49.917447 21693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:50.030589 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 21631
2026-06-19T18:45:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4631 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T18:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:50Z Disabled control of system clock
I20260619 18:45:50.111395 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37615
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:50.473318 22174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:50.473713 22174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:50.473873 22174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:50.481570 22174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:50.481760 22174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:50.481868 22174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:50.481973 22174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:50.482052 22174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:50.491919 22174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:50.493356 22174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:50.495365 22174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:50.508414 22179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:50.508314 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:50.508381 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:50.509514 22174 server_base.cc:1034] running on GCE node
I20260619 18:45:50.510351 22174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:50.512024 22174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:50.513296 22174 hybrid_clock.cc:648] HybridClock initialized: now 1781894750513266 us; error 70 us; skew 500 ppm
I20260619 18:45:50.513695 22174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:50.516285 22174 webserver.cc:460] Webserver started at http://127.3.85.190:44709/ using document root <none> and password file <none>
I20260619 18:45:50.517059 22174 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:50.517205 22174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:50.517558 22174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:50.520805 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "ce6546c225974e229e326cf82372f2a4"
format_stamp: "Formatted at 2026-06-19 18:45:50 on dist-test-slave-09fp"
server_key: "5d167b5b9932a592e856610579468753"
server_key_iv: "31cff691e35baa73a6d6c796372ea310"
server_key_version: "encryptionkey@0"
I20260619 18:45:50.521708 22174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "ce6546c225974e229e326cf82372f2a4"
format_stamp: "Formatted at 2026-06-19 18:45:50 on dist-test-slave-09fp"
server_key: "5d167b5b9932a592e856610579468753"
server_key_iv: "31cff691e35baa73a6d6c796372ea310"
server_key_version: "encryptionkey@0"
I20260619 18:45:50.527534 22174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 18:45:50.531355 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:50.532800 22174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:50.533023 22174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "ce6546c225974e229e326cf82372f2a4"
format_stamp: "Formatted at 2026-06-19 18:45:50 on dist-test-slave-09fp"
server_key: "5d167b5b9932a592e856610579468753"
server_key_iv: "31cff691e35baa73a6d6c796372ea310"
server_key_version: "encryptionkey@0"
I20260619 18:45:50.533298 22174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:50.548439 22174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:50.549381 22174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:50.549708 22174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:50.570174 22174 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37615
I20260619 18:45:50.570209 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37615 every 8 connection(s)
I20260619 18:45:50.571713 22174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:50.576789 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:50.581533 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22174
I20260619 18:45:50.581755 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:50.582338 3414 external_mini_cluster.cc:1428] Setting key 773c5171b3188fb8c27c4b2f536cad79
I20260619 18:45:50.591554 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: Bootstrap starting.
I20260619 18:45:50.595598 22240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:50.596933 22240 log.cc:824] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:50.597255 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:50.584585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.3.85.190:37615)
0619 18:45:50.585703 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:50.585714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:50.585758 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:45:50.587567 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:50.587595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:50.588158 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:50.588594 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:50.589237 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:50.590616 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:50.592088 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:50.592849 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:50.593084 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:50.594430 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:50.594439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:50.594509 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:50.595126 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:50.595233 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:50.595263 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:50.595418 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:50.595705 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:45:50.596141 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":419,"threads_started":1}
I20260619 18:45:50.600651 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: No bootstrap required, opened a new log
I20260619 18:45:50.609149 22240 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } }
I20260619 18:45:50.609602 22240 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:50.609679 22240 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6546c225974e229e326cf82372f2a4, State: Initialized, Role: FOLLOWER
I20260619 18:45:50.610435 22240 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } }
I20260619 18:45:50.610811 22240 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:50.610981 22240 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:50.611212 22240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:50.613991 22240 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } }
I20260619 18:45:50.614694 22240 leader_election.cc:302] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce6546c225974e229e326cf82372f2a4; no voters:
I20260619 18:45:50.615325 22240 leader_election.cc:288] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:50.615535 22245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:50.616371 22245 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [term 1 LEADER]: Becoming Leader. State: Replica: ce6546c225974e229e326cf82372f2a4, State: Running, Role: LEADER
I20260619 18:45:50.617018 22245 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } }
I20260619 18:45:50.617854 22240 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:50.620589 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce6546c225974e229e326cf82372f2a4. Latest consensus state: current_term: 1 leader_uuid: "ce6546c225974e229e326cf82372f2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } } }
I20260619 18:45:50.620755 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce6546c225974e229e326cf82372f2a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6546c225974e229e326cf82372f2a4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37615 } } }
I20260619 18:45:50.621505 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:50.621541 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:50.628211 22252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:50.634214 22252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:50.644924 22252 catalog_manager.cc:1347] Generated new cluster ID: d6bec8bbc7764845b776640ed6f6acea
I20260619 18:45:50.645076 22252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:50.671383 22252 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:50.673049 22252 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:50.681455 22252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: Generated new TSK 0
I20260619 18:45:50.682412 22252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:45:50.690336 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--builtin_ntp_servers=127.3.85.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:51.074994 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:51.075307 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:51.075398 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:51.082926 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:51.083047 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:51.083158 22264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:51.092617 22264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:51.093951 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:51.095901 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:51.110810 22269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:51.110894 22272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:51.110894 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:51.111903 22264 server_base.cc:1034] running on GCE node
I20260619 18:45:51.112622 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:51.113770 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:51.115085 22264 hybrid_clock.cc:648] HybridClock initialized: now 1781894751115038 us; error 39 us; skew 500 ppm
I20260619 18:45:51.115449 22264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:51.117976 22264 webserver.cc:460] Webserver started at http://127.3.85.129:39735/ using document root <none> and password file <none>
I20260619 18:45:51.118871 22264 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:51.118997 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:51.119395 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:51.122609 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "390145e6f60c4d7d9f8676892b127ff1"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "378657c368158a02de9ac24238306fbb"
server_key_iv: "d29cff1ac33c15b92b666120756944b0"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.123602 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "390145e6f60c4d7d9f8676892b127ff1"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "378657c368158a02de9ac24238306fbb"
server_key_iv: "d29cff1ac33c15b92b666120756944b0"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.130098 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:45:51.134264 22278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.135985 22264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:51.136222 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "390145e6f60c4d7d9f8676892b127ff1"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "378657c368158a02de9ac24238306fbb"
server_key_iv: "d29cff1ac33c15b92b666120756944b0"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.136540 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:51.152119 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:51.153005 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:51.153293 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:51.154266 22264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:51.156061 22264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:51.156162 22264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.156255 22264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:51.156311 22264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.190781 22264 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:34001
I20260619 18:45:51.190953 22390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:34001 every 8 connection(s)
I20260619 18:45:51.192287 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:51.193005 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22264
I20260619 18:45:51.193203 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:51.193782 3414 external_mini_cluster.cc:1428] Setting key 1dac7de9423fa028f4b0e868121a4591
I20260619 18:45:51.197203 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--builtin_ntp_servers=127.3.85.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:51.209364 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:51.195627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:37603 (local address 127.3.85.190:37615)
0619 18:45:51.198973 (+ 3346us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:51.198982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:51.199009 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:45:51.200597 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:51.200607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:51.200712 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:51.200846 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:51.202200 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.203457 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.205333 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.206142 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.206332 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:51.208393 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:51.208398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:51.208402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:51.208425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:51.208469 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:51.208482 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:51.208596 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:51.208870 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 18:45:51.208991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3109,"thread_start_us":2968,"threads_started":1}
I20260619 18:45:51.210602 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:51.196348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37615 (local address 127.3.85.129:37603)
0619 18:45:51.197893 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:51.197962 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:51.200220 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:51.201137 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:51.201147 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:51.201259 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:51.201953 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:51.201974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.203690 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.203695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:51.205117 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:51.205136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.206730 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.206752 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:51.207047 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:51.207944 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:51.207947 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:51.207998 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:51.208100 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:51.208599 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:51.208614 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:51.208627 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:45:51.208849 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 18:45:51.209357 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":445,"threads_started":1}
I20260619 18:45:51.212837 22391 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37615
I20260619 18:45:51.213230 22391 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:51.214253 22391 heartbeater.cc:511] Master 127.3.85.190:37615 requested a full tablet report, sending...
I20260619 18:45:51.217255 22205 ts_manager.cc:194] Registered new tserver with Master: 390145e6f60c4d7d9f8676892b127ff1 (127.3.85.129:34001)
I20260619 18:45:51.220033 22205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.129:37603
W20260619 18:45:51.550946 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:51.551254 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:51.551343 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:51.559073 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:51.559194 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:51.559305 22395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:51.569573 22395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:51.571249 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:51.573683 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:51.590911 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:51.590950 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:51.596415 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:51.597357 22395 server_base.cc:1034] running on GCE node
I20260619 18:45:51.598090 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:51.599336 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:51.600594 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781894751600493 us; error 104 us; skew 500 ppm
I20260619 18:45:51.600987 22395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:51.603739 22395 webserver.cc:460] Webserver started at http://127.3.85.130:46851/ using document root <none> and password file <none>
I20260619 18:45:51.604491 22395 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:51.604596 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:51.604903 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:51.608182 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "f32f776970094fd19b20a8dc10e1915c"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "67c9d83ce2476dad52250374650fcdb5"
server_key_iv: "7e217f57965a4b0b60f611bea3c57f6e"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.609041 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "f32f776970094fd19b20a8dc10e1915c"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "67c9d83ce2476dad52250374650fcdb5"
server_key_iv: "7e217f57965a4b0b60f611bea3c57f6e"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.616590 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260619 18:45:51.620883 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.622607 22395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:51.622815 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "f32f776970094fd19b20a8dc10e1915c"
format_stamp: "Formatted at 2026-06-19 18:45:51 on dist-test-slave-09fp"
server_key: "67c9d83ce2476dad52250374650fcdb5"
server_key_iv: "7e217f57965a4b0b60f611bea3c57f6e"
server_key_version: "encryptionkey@0"
I20260619 18:45:51.623039 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:51.656301 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:51.657213 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:51.657490 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:51.658563 22395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:51.660132 22395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:51.660229 22395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.660308 22395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:51.660363 22395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:51.696537 22395 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:41511
I20260619 18:45:51.696585 22521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:41511 every 8 connection(s)
I20260619 18:45:51.697976 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:51.708379 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22395
I20260619 18:45:51.708581 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:51.709056 3414 external_mini_cluster.cc:1428] Setting key 4de3f216c86d4787780f295e4f25e79f
I20260619 18:45:51.712350 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--builtin_ntp_servers=127.3.85.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:51.712622 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:51.700857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:52451 (local address 127.3.85.190:37615)
0619 18:45:51.701099 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:51.701106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:51.702690 (+ 1584us) server_negotiation.cc:407] Connection header received
0619 18:45:51.704738 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:51.704742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:51.704854 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:51.705009 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:51.706396 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.707222 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.709154 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.709667 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.709825 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:51.711695 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:51.711701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:51.711704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:51.711726 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:51.711774 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:51.711791 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:51.711905 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:51.712163 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 18:45:51.712275 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:45:51.713778 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:51.701412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37615 (local address 127.3.85.130:52451)
0619 18:45:51.702418 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:51.702547 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:51.704368 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:51.705286 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:51.705296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:51.705398 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:51.706185 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:51.706208 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.707418 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.707423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:51.708963 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:51.708984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:51.709978 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:51.709989 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:51.710317 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:51.711231 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:51.711235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:51.711297 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:51.711418 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:51.711906 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:51.711920 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:51.711931 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:45:51.712151 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:45:51.712639 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260619 18:45:51.715462 22522 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37615
I20260619 18:45:51.715806 22522 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:51.716612 22522 heartbeater.cc:511] Master 127.3.85.190:37615 requested a full tablet report, sending...
I20260619 18:45:51.718309 22205 ts_manager.cc:194] Registered new tserver with Master: f32f776970094fd19b20a8dc10e1915c (127.3.85.130:41511)
I20260619 18:45:51.719381 22205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.130:52451
W20260619 18:45:52.081447 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:52.081771 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:52.081923 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:52.089512 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:52.089635 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:52.089818 22525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:52.100757 22525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:37615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:52.102262 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:52.104192 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:52.119040 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:52.119017 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:52.118988 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:52.119920 22525 server_base.cc:1034] running on GCE node
I20260619 18:45:52.120664 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:52.121661 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:52.122875 22525 hybrid_clock.cc:648] HybridClock initialized: now 1781894752122830 us; error 41 us; skew 500 ppm
I20260619 18:45:52.123188 22525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:52.125559 22525 webserver.cc:460] Webserver started at http://127.3.85.131:38157/ using document root <none> and password file <none>
I20260619 18:45:52.126289 22525 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:52.126386 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:52.126773 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:52.130118 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "07854b3ceee94e4d9026c81592a0dbe0"
format_stamp: "Formatted at 2026-06-19 18:45:52 on dist-test-slave-09fp"
server_key: "d5f2639c7ea38907c5b41303ba00cca1"
server_key_iv: "dc33ed0c2811299009544a5048952212"
server_key_version: "encryptionkey@0"
I20260619 18:45:52.131040 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "07854b3ceee94e4d9026c81592a0dbe0"
format_stamp: "Formatted at 2026-06-19 18:45:52 on dist-test-slave-09fp"
server_key: "d5f2639c7ea38907c5b41303ba00cca1"
server_key_iv: "dc33ed0c2811299009544a5048952212"
server_key_version: "encryptionkey@0"
I20260619 18:45:52.136896 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:45:52.141278 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:52.143115 22525 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:45:52.143287 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "07854b3ceee94e4d9026c81592a0dbe0"
format_stamp: "Formatted at 2026-06-19 18:45:52 on dist-test-slave-09fp"
server_key: "d5f2639c7ea38907c5b41303ba00cca1"
server_key_iv: "dc33ed0c2811299009544a5048952212"
server_key_version: "encryptionkey@0"
I20260619 18:45:52.143499 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:52.167786 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:52.168642 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:52.168907 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:52.169958 22525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:52.171609 22525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:52.171701 22525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:52.171777 22525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:52.171834 22525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:52.209472 22525 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:32787
I20260619 18:45:52.209596 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:32787 every 8 connection(s)
I20260619 18:45:52.211061 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:52.221439 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22525
I20260619 18:45:52.221709 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:52.222563 3414 external_mini_cluster.cc:1428] Setting key ffd849b65489a32def9e3929902ae68b
I20260619 18:45:52.223881 22391 heartbeater.cc:503] Master 127.3.85.190:37615 was elected leader, sending a full tablet report...
I20260619 18:45:52.230309 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.214144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:34901 (local address 127.3.85.190:37615)
0619 18:45:52.214706 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.214713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.216643 (+ 1930us) server_negotiation.cc:407] Connection header received
0619 18:45:52.219559 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.219574 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.219708 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.219830 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:52.221273 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.222869 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.225583 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.226188 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.226313 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.229008 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:52.229014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:52.229018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:52.229039 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:52.229085 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:52.229099 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:52.229184 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.229894 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.230000 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":302,"threads_started":1}
I20260619 18:45:52.231613 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.214862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:37615 (local address 127.3.85.131:34901)
0619 18:45:52.216321 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.216390 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.218953 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.220126 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.220137 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.220270 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:52.220956 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.220976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.223167 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.223183 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.225407 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.225428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.226754 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.226764 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.227099 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.228511 (+ 1412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:52.228515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:52.228596 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:52.228722 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:52.229336 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:52.229353 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:52.229365 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.229829 (+ 464us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.230240 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":458,"threads_started":1}
I20260619 18:45:52.233471 22652 heartbeater.cc:347] Connected to a master server at 127.3.85.190:37615
I20260619 18:45:52.233843 22652 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:52.234791 22652 heartbeater.cc:511] Master 127.3.85.190:37615 requested a full tablet report, sending...
I20260619 18:45:52.237133 22204 ts_manager.cc:194] Registered new tserver with Master: 07854b3ceee94e4d9026c81592a0dbe0 (127.3.85.131:32787)
I20260619 18:45:52.238397 22204 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.131:34901
I20260619 18:45:52.239930 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:52.253226 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.245703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.3.85.190:37615)
0619 18:45:52.245944 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.245949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.246568 (+ 619us) server_negotiation.cc:407] Connection header received
0619 18:45:52.246865 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.246872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.246967 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.247077 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:52.247786 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.248605 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.250098 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.250655 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.250771 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.252358 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:52.252365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:52.252370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:52.252499 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:52.252547 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:52.252571 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:52.252756 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.252928 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.252980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 18:45:52.258983 22204 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:52.266095 22204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:52.301633 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.281406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:41511 (local address 127.0.0.1:43682)
0619 18:45:52.291149 (+ 9743us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.291202 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.291433 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.292922 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.292931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.292965 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:52.293588 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.293603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.295575 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.295592 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.298169 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.298190 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.300539 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.300547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.300789 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.300865 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.301010 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.301249 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9445,"thread_start_us":11171,"threads_started":1}
I20260619 18:45:52.302050 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.282392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.3.85.130:41511)
0619 18:45:52.289906 (+ 7514us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.289919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.291653 (+ 1734us) server_negotiation.cc:407] Connection header received
0619 18:45:52.291778 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.291788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.292045 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.292285 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:52.293793 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.295332 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.298417 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.299852 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.300023 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.300342 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.301466 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.301710 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7252,"thread_start_us":7028,"threads_started":1}
I20260619 18:45:52.303421 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.281838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:32787 (local address 127.0.0.1:57022)
0619 18:45:52.293459 (+ 11621us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.293479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.293608 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.294537 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.294547 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.294587 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:52.295165 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.295182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.297443 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.297448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.299874 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.299890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.301959 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.301966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.302136 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.302168 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.302861 (+ 693us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.303021 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11476,"mutex_wait_us":58,"thread_start_us":8855,"threads_started":1}
I20260619 18:45:52.303539 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.281501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.3.85.129:34001)
0619 18:45:52.292959 (+ 11458us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.292967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.293142 (+ 175us) server_negotiation.cc:407] Connection header received
0619 18:45:52.293681 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.293690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.293960 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.294196 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:52.294928 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.296712 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.300120 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.301667 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.301976 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.302338 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.302939 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.303149 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11271,"thread_start_us":11679,"threads_started":1}
I20260619 18:45:52.303489 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.282817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.3.85.131:32787)
0619 18:45:52.290391 (+ 7574us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.290404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.293686 (+ 3282us) server_negotiation.cc:407] Connection header received
0619 18:45:52.293789 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.293797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.293993 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.294196 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:52.295413 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.297271 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.300100 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.301667 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.302402 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.302727 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.302923 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.303122 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7337,"thread_start_us":7122,"threads_started":1}
I20260619 18:45:52.304303 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.281468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50222)
0619 18:45:52.292945 (+ 11477us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.292974 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.293464 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.294200 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.294205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.294232 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:52.294691 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.294704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.297142 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.297146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.299809 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.299833 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.302559 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.302564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.302735 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.302767 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.303873 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.304044 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11262,"mutex_wait_us":164,"thread_start_us":11674,"threads_started":1}
I20260619 18:45:52.306643 22457 tablet_service.cc:1467] Processing CreateTablet for tablet acec4767dda946d8808bbe93c9de6b90 (DEFAULT_TABLE table=test-table [id=047fac5a81a64448b17ef822f2093765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:52.308306 22457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acec4767dda946d8808bbe93c9de6b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:52.307734 22587 tablet_service.cc:1467] Processing CreateTablet for tablet acec4767dda946d8808bbe93c9de6b90 (DEFAULT_TABLE table=test-table [id=047fac5a81a64448b17ef822f2093765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:52.308784 22326 tablet_service.cc:1467] Processing CreateTablet for tablet acec4767dda946d8808bbe93c9de6b90 (DEFAULT_TABLE table=test-table [id=047fac5a81a64448b17ef822f2093765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:52.309257 22587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acec4767dda946d8808bbe93c9de6b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:52.309998 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acec4767dda946d8808bbe93c9de6b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:52.324141 22671 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Bootstrap starting.
I20260619 18:45:52.327034 22672 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Bootstrap starting.
I20260619 18:45:52.328164 22673 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Bootstrap starting.
I20260619 18:45:52.330148 22671 tablet_bootstrap.cc:654] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:52.331697 22672 tablet_bootstrap.cc:654] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:52.331842 22671 log.cc:824] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:52.332583 22673 tablet_bootstrap.cc:654] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:52.333173 22672 log.cc:824] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:52.333762 22673 log.cc:824] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:52.335964 22671 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: No bootstrap required, opened a new log
I20260619 18:45:52.336391 22671 ts_tablet_manager.cc:1397] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 18:45:52.342365 22673 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: No bootstrap required, opened a new log
I20260619 18:45:52.342970 22673 ts_tablet_manager.cc:1397] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260619 18:45:52.343220 22672 tablet_bootstrap.cc:492] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: No bootstrap required, opened a new log
I20260619 18:45:52.343514 22672 ts_tablet_manager.cc:1397] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260619 18:45:52.350744 22671 raft_consensus.cc:348] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.351328 22671 raft_consensus.cc:374] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:52.351508 22671 raft_consensus.cc:729] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390145e6f60c4d7d9f8676892b127ff1, State: Initialized, Role: FOLLOWER
I20260619 18:45:52.351836 22672 raft_consensus.cc:348] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.352317 22672 raft_consensus.cc:374] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:52.352435 22672 raft_consensus.cc:729] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32f776970094fd19b20a8dc10e1915c, State: Initialized, Role: FOLLOWER
I20260619 18:45:52.352332 22671 consensus_queue.cc:260] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.353067 22672 consensus_queue.cc:260] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.354602 22671 ts_tablet_manager.cc:1428] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260619 18:45:52.355902 22522 heartbeater.cc:503] Master 127.3.85.190:37615 was elected leader, sending a full tablet report...
I20260619 18:45:52.355345 22673 raft_consensus.cc:348] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.356598 22672 ts_tablet_manager.cc:1428] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260619 18:45:52.356735 22673 raft_consensus.cc:374] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:52.356870 22673 raft_consensus.cc:729] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07854b3ceee94e4d9026c81592a0dbe0, State: Initialized, Role: FOLLOWER
I20260619 18:45:52.357564 22673 consensus_queue.cc:260] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.358875 22652 heartbeater.cc:503] Master 127.3.85.190:37615 was elected leader, sending a full tablet report...
I20260619 18:45:52.360181 22673 ts_tablet_manager.cc:1428] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260619 18:45:52.383445 22678 raft_consensus.cc:482] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:52.383895 22678 raft_consensus.cc:504] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.385907 22678 leader_election.cc:288] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 390145e6f60c4d7d9f8676892b127ff1 (127.3.85.129:34001), 07854b3ceee94e4d9026c81592a0dbe0 (127.3.85.131:32787)
I20260619 18:45:52.396319 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.386695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:34349 (local address 127.3.85.129:34001)
0619 18:45:52.386938 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.386947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.387591 (+ 644us) server_negotiation.cc:407] Connection header received
0619 18:45:52.387845 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.387851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.387940 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.388052 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:52.389186 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.390252 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.393597 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.394953 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.395153 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.395223 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.395895 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.396043 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:45:52.396602 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.386175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:32787 (local address 127.3.85.130:49501)
0619 18:45:52.387366 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.387391 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.387605 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.388121 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.388126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.388185 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:52.388737 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.388761 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.390442 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.390475 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.393221 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.393243 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.395256 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.395262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.395559 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.395723 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.395836 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.396078 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":311,"spinlock_wait_cycles":31616,"thread_start_us":801,"threads_started":1}
I20260619 18:45:52.397130 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.386266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.3.85.130:34349)
0619 18:45:52.387484 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.387501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:52.387661 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:52.388128 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:52.388131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:52.388159 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:52.388945 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.388976 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.390441 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.390475 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.393221 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:52.393243 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.395254 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.395262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:52.395556 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.395726 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 18:45:52.395836 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:45:52.396078 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":359,"spinlock_wait_cycles":23808,"thread_start_us":882,"threads_started":1}
I20260619 18:45:52.397138 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.386664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:49501 (local address 127.3.85.131:32787)
0619 18:45:52.386963 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.386973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.387514 (+ 541us) server_negotiation.cc:407] Connection header received
0619 18:45:52.387845 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.387852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.387939 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.388052 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:52.389103 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.390261 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.393596 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.394953 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.395148 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.395223 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.396687 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.396847 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:45:52.398630 22607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acec4767dda946d8808bbe93c9de6b90" candidate_uuid: "f32f776970094fd19b20a8dc10e1915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07854b3ceee94e4d9026c81592a0dbe0" is_pre_election: true
I20260619 18:45:52.398667 22346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acec4767dda946d8808bbe93c9de6b90" candidate_uuid: "f32f776970094fd19b20a8dc10e1915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390145e6f60c4d7d9f8676892b127ff1" is_pre_election: true
I20260619 18:45:52.399206 22346 raft_consensus.cc:2454] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32f776970094fd19b20a8dc10e1915c in term 0.
I20260619 18:45:52.399204 22607 raft_consensus.cc:2454] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32f776970094fd19b20a8dc10e1915c in term 0.
I20260619 18:45:52.399977 22411 leader_election.cc:302] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07854b3ceee94e4d9026c81592a0dbe0, 390145e6f60c4d7d9f8676892b127ff1, f32f776970094fd19b20a8dc10e1915c; no voters:
I20260619 18:45:52.400520 22678 raft_consensus.cc:2781] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:52.400715 22678 raft_consensus.cc:482] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:52.400831 22678 raft_consensus.cc:3037] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:52.403373 22678 raft_consensus.cc:504] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.404038 22678 leader_election.cc:288] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [CANDIDATE]: Term 1 election: Requested vote from peers 390145e6f60c4d7d9f8676892b127ff1 (127.3.85.129:34001), 07854b3ceee94e4d9026c81592a0dbe0 (127.3.85.131:32787)
I20260619 18:45:52.404734 22346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acec4767dda946d8808bbe93c9de6b90" candidate_uuid: "f32f776970094fd19b20a8dc10e1915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390145e6f60c4d7d9f8676892b127ff1"
I20260619 18:45:52.404764 22607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acec4767dda946d8808bbe93c9de6b90" candidate_uuid: "f32f776970094fd19b20a8dc10e1915c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07854b3ceee94e4d9026c81592a0dbe0"
I20260619 18:45:52.404990 22607 raft_consensus.cc:3037] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:52.404991 22346 raft_consensus.cc:3037] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:52.408483 22346 raft_consensus.cc:2454] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32f776970094fd19b20a8dc10e1915c in term 1.
I20260619 18:45:52.408483 22607 raft_consensus.cc:2454] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32f776970094fd19b20a8dc10e1915c in term 1.
I20260619 18:45:52.409123 22411 leader_election.cc:302] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 390145e6f60c4d7d9f8676892b127ff1, f32f776970094fd19b20a8dc10e1915c; no voters:
I20260619 18:45:52.409695 22678 raft_consensus.cc:2781] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:52.410564 22678 raft_consensus.cc:686] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 1 LEADER]: Becoming Leader. State: Replica: f32f776970094fd19b20a8dc10e1915c, State: Running, Role: LEADER
I20260619 18:45:52.411181 22678 consensus_queue.cc:237] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } }
I20260619 18:45:52.416152 22205 catalog_manager.cc:5495] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c reported cstate change: term changed from 0 to 1, leader changed from <none> to f32f776970094fd19b20a8dc10e1915c (127.3.85.130). New cstate: current_term: 1 leader_uuid: "f32f776970094fd19b20a8dc10e1915c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f32f776970094fd19b20a8dc10e1915c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:52.434558 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.428643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.3.85.130:41511)
0619 18:45:52.428877 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.428882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.428922 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:45:52.429145 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.429150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.429248 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.429386 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:52.429859 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.430713 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.432088 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.432524 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.432630 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.433183 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:52.433708 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:52.433870 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.434136 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.434216 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:45:52.444320 22607 raft_consensus.cc:1261] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 1 FOLLOWER]: Refusing update from remote peer f32f776970094fd19b20a8dc10e1915c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:52.444319 22346 raft_consensus.cc:1261] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 1 FOLLOWER]: Refusing update from remote peer f32f776970094fd19b20a8dc10e1915c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 18:45:52.445174 22392 tablet.cc:2369] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:52.445528 22682 consensus_queue.cc:1035] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [LEADER]: Connected to new peer: Peer: permanent_uuid: "07854b3ceee94e4d9026c81592a0dbe0" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:52.445895 22678 consensus_queue.cc:1035] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [LEADER]: Connected to new peer: Peer: permanent_uuid: "390145e6f60c4d7d9f8676892b127ff1" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:45:52.450331 22523 tablet.cc:2369] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:52.474957 22203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:52.475320 22203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:52.480034 22203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce6546c225974e229e326cf82372f2a4: Sending DeleteTablet for 3 replicas of tablet acec4767dda946d8808bbe93c9de6b90
I20260619 18:45:52.481433 22326 tablet_service.cc:1514] Processing DeleteTablet for tablet acec4767dda946d8808bbe93c9de6b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:52 UTC) from {username='slave'} at 127.0.0.1:50222
I20260619 18:45:52.481438 22457 tablet_service.cc:1514] Processing DeleteTablet for tablet acec4767dda946d8808bbe93c9de6b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:52 UTC) from {username='slave'} at 127.0.0.1:43682
I20260619 18:45:52.481805 22587 tablet_service.cc:1514] Processing DeleteTablet for tablet acec4767dda946d8808bbe93c9de6b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:52 UTC) from {username='slave'} at 127.0.0.1:57022
I20260619 18:45:52.486048 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22264
I20260619 18:45:52.493898 22694 tablet_replica.cc:330] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: stopping tablet replica
I20260619 18:45:52.493898 22696 tablet_replica.cc:330] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: stopping tablet replica
I20260619 18:45:52.494424 22696 raft_consensus.cc:2229] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:52.494424 22694 raft_consensus.cc:2229] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:52.495147 22694 raft_consensus.cc:2258] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:52.496263 22695 tablet_replica.cc:330] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: stopping tablet replica
I20260619 18:45:52.496301 22696 raft_consensus.cc:2258] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:52.496929 22695 raft_consensus.cc:2229] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:52.497640 22695 raft_consensus.cc:2258] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:52.498893 22696 ts_tablet_manager.cc:1905] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:52.499291 22694 ts_tablet_manager.cc:1905] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:52.508200 22696 ts_tablet_manager.cc:1918] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:52.508488 22696 log.cc:1196] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/acec4767dda946d8808bbe93c9de6b90
I20260619 18:45:52.508637 22695 ts_tablet_manager.cc:1905] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:52.509092 22696 ts_tablet_manager.cc:1939] T acec4767dda946d8808bbe93c9de6b90 P 07854b3ceee94e4d9026c81592a0dbe0: Deleting consensus metadata
I20260619 18:45:52.510876 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.487510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.3.85.129:34001)
0619 18:45:52.487712 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.487720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.501145 (+ 13425us) server_negotiation.cc:407] Connection header received
0619 18:45:52.501710 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.501717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.501833 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.501922 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:52.502707 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.503887 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.506102 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.506916 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.507055 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.508829 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:52.508842 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:52.508861 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:52.509759 (+ 898us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:52.509830 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:52.509859 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:52.509962 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.510259 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.510402 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:45:52.511200 22191 catalog_manager.cc:4841] TS 07854b3ceee94e4d9026c81592a0dbe0 (127.3.85.131:32787): tablet acec4767dda946d8808bbe93c9de6b90 (table test-table [id=047fac5a81a64448b17ef822f2093765]) successfully deleted
I20260619 18:45:52.512346 22386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:52.513052 22694 ts_tablet_manager.cc:1918] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:52.513342 22694 log.cc:1196] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/acec4767dda946d8808bbe93c9de6b90
I20260619 18:45:52.517545 22695 ts_tablet_manager.cc:1918] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:52.517778 22695 log.cc:1196] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/acec4767dda946d8808bbe93c9de6b90
I20260619 18:45:52.518347 22695 ts_tablet_manager.cc:1939] T acec4767dda946d8808bbe93c9de6b90 P f32f776970094fd19b20a8dc10e1915c: Deleting consensus metadata
I20260619 18:45:52.520084 22192 catalog_manager.cc:4841] TS f32f776970094fd19b20a8dc10e1915c (127.3.85.130:41511): tablet acec4767dda946d8808bbe93c9de6b90 (table test-table [id=047fac5a81a64448b17ef822f2093765]) successfully deleted
I20260619 18:45:52.652791 22694 ts_tablet_manager.cc:1939] T acec4767dda946d8808bbe93c9de6b90 P 390145e6f60c4d7d9f8676892b127ff1: Deleting consensus metadata
I20260619 18:45:52.652974 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22264
W20260619 18:45:52.678160 22190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:34001 (error 108)
I20260619 18:45:52.678332 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22395
I20260619 18:45:52.680538 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.680029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50240)
0619 18:45:52.680217 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.680342 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 18:45:52.681839 22190 catalog_manager.cc:4584] TS 390145e6f60c4d7d9f8676892b127ff1 (127.3.85.129:34001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acec4767dda946d8808bbe93c9de6b90: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
I20260619 18:45:52.686645 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.679619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.3.85.130:41511)
0619 18:45:52.679803 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.679809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.679829 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:45:52.679906 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.679912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.680032 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.680147 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:52.680764 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.681532 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.683208 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.683664 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.683823 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.685073 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:52.685084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:52.685100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:52.685750 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:52.685800 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:52.685825 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:52.685962 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.686109 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.686216 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:45:52.687414 22517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:52.735683 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.735099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50252)
0619 18:45:52.735355 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.735473 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 18:45:52.736969 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.736532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50256)
0619 18:45:52.736733 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.736823 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:45:52.797712 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.797188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50268)
0619 18:45:52.797446 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.797543 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 18:45:52.799090 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.798669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50272)
0619 18:45:52.798853 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.798935 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:45:52.805466 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22395
I20260619 18:45:52.825481 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22525
I20260619 18:45:52.833084 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.826494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.3.85.131:32787)
0619 18:45:52.826699 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:52.826704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:52.826823 (+ 119us) server_negotiation.cc:407] Connection header received
0619 18:45:52.826989 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:52.826993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:52.827086 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:52.827175 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:52.827541 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.828366 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.829951 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:52.830353 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:52.830614 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:52.831671 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:52.831680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:52.831695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:52.832304 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:52.832362 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:52.832384 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:52.832475 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:52.832604 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 18:45:52.832717 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:45:52.833740 22647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:52.879944 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.879352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50276)
0619 18:45:52.879617 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.879749 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:45:52.881232 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:52.880720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50286)
0619 18:45:52.880967 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:52.881063 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 18:45:52.928543 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22525
I20260619 18:45:52.946287 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22174
I20260619 18:45:52.947638 22235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:53.059605 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:53.058904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:34001 (local address 127.0.0.1:50298)
0619 18:45:53.059118 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:53.059252 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:45:53.059926 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22174
2026-06-19T18:45:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:45:53 dist-test-slave-09fp krb5kdc[22705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:45:53 dist-test-slave-09fp krb5kdc[22705](info): set up 2 sockets
Jun 19 18:45:53 dist-test-slave-09fp krb5kdc[22705](info): commencing operation
krb5kdc: starting...
W20260619 18:45:54.199141 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:45:54 dist-test-slave-09fp krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:45:54.483227 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39937
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:54.843636 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:54.844038 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:54.844221 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:54.853879 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:54.854070 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:54.854184 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:54.854291 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:54.854380 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:54.865149 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:54.866744 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:54.868147 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:54.881754 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:54.881779 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:54.881834 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:54.882758 22718 server_base.cc:1034] running on GCE node
I20260619 18:45:54.883608 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:54.885290 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:54.886673 22718 hybrid_clock.cc:648] HybridClock initialized: now 1781894754886619 us; error 42 us; skew 500 ppm
I20260619 18:45:54.887041 22718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:54 dist-test-slave-09fp krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:54.890635 22718 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:45:54.892369 22718 webserver.cc:460] Webserver started at http://127.3.85.190:45623/ using document root <none> and password file <none>
I20260619 18:45:54.893138 22718 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:54.893252 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:54.893530 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:54.896894 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "161e82bd06774c96858e757d9129d83b"
format_stamp: "Formatted at 2026-06-19 18:45:54 on dist-test-slave-09fp"
server_key: "4bbaa193d72a8b1346bec11625137dc4"
server_key_iv: "bab039bcc81badf5b5efd4f2063b480b"
server_key_version: "encryptionkey@0"
I20260619 18:45:54.897739 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "161e82bd06774c96858e757d9129d83b"
format_stamp: "Formatted at 2026-06-19 18:45:54 on dist-test-slave-09fp"
server_key: "4bbaa193d72a8b1346bec11625137dc4"
server_key_iv: "bab039bcc81badf5b5efd4f2063b480b"
server_key_version: "encryptionkey@0"
I20260619 18:45:54.904289 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:45:54.908663 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:54.910360 22718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:45:54.910648 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "161e82bd06774c96858e757d9129d83b"
format_stamp: "Formatted at 2026-06-19 18:45:54 on dist-test-slave-09fp"
server_key: "4bbaa193d72a8b1346bec11625137dc4"
server_key_iv: "bab039bcc81badf5b5efd4f2063b480b"
server_key_version: "encryptionkey@0"
I20260619 18:45:54.910887 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:54.926604 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:54.931972 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:54.932286 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:54.954275 22718 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39937
I20260619 18:45:54.954296 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39937 every 8 connection(s)
I20260619 18:45:54.955916 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:54.960361 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22718
I20260619 18:45:54.960367 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:54.960605 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:54.961056 3414 external_mini_cluster.cc:1428] Setting key 61908bb9fd00a1396c94eb3c0f3957ee
I20260619 18:45:54.977945 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: Bootstrap starting.
Jun 19 18:45:54 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:54.982627 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:54.984258 22785 log.cc:824] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:54.988181 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: No bootstrap required, opened a new log
I20260619 18:45:54.990003 22788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:54.963072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.3.85.190:39937)
0619 18:45:54.964095 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:54.964113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:54.964174 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:45:54.966300 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:54.966341 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:54.967060 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:54.967598 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:54.968992 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:54.971353 (+ 2361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:54.972875 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:54.974096 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:54.974345 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:54.979661 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:54.979677 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:54.979708 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:54.979791 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:54.984293 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:54.985481 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:54.985492 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:54.985499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:54.985704 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:54.986142 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:54.986147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:54.986150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:54.986932 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:54.987283 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:54.988024 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 18:45:54.988534 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":409,"threads_started":1}
I20260619 18:45:54.997822 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } }
I20260619 18:45:54.998261 22785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:54.998339 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161e82bd06774c96858e757d9129d83b, State: Initialized, Role: FOLLOWER
I20260619 18:45:54.998968 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } }
I20260619 18:45:54.999215 22785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:54.999382 22785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:54.999569 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:55.002278 22785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } }
I20260619 18:45:55.002940 22785 leader_election.cc:302] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161e82bd06774c96858e757d9129d83b; no voters:
I20260619 18:45:55.003538 22785 leader_election.cc:288] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:55.003731 22790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:55.004995 22790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [term 1 LEADER]: Becoming Leader. State: Replica: 161e82bd06774c96858e757d9129d83b, State: Running, Role: LEADER
I20260619 18:45:55.005666 22790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } }
I20260619 18:45:55.006106 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:55.008955 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161e82bd06774c96858e757d9129d83b. Latest consensus state: current_term: 1 leader_uuid: "161e82bd06774c96858e757d9129d83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } } }
I20260619 18:45:55.009212 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161e82bd06774c96858e757d9129d83b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161e82bd06774c96858e757d9129d83b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39937 } } }
I20260619 18:45:55.009498 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:55.009531 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:55.012125 22799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:55.017365 22799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:55.027204 22799 catalog_manager.cc:1347] Generated new cluster ID: d33d090edcd74e03afc188f95ed7840a
I20260619 18:45:55.027351 22799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:55.039248 22799 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:55.041289 22799 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:55.054181 22799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: Generated new TSK 0
I20260619 18:45:55.055450 22799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.85.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
Entry for principal kudu/127.3.85.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab.
I20260619 18:45:55.126897 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:55.486804 22811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:55.487110 22811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:55.487272 22811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:55.495486 22811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:55.495628 22811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:55.495839 22811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:55.506558 22811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.129.keytab
--principal=kudu/127.3.85.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:55.508066 22811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:55.509605 22811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:55.524427 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:55.524605 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:55.524510 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:55.525893 22811 server_base.cc:1034] running on GCE node
I20260619 18:45:55.526669 22811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:55.527745 22811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:55.529011 22811 hybrid_clock.cc:648] HybridClock initialized: now 1781894755528954 us; error 41 us; skew 500 ppm
I20260619 18:45:55.529428 22811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:55 dist-test-slave-09fp krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:55.533206 22811 init.cc:377] Logged in from keytab as kudu/127.3.85.129@KRBTEST.COM (short username kudu)
I20260619 18:45:55.535154 22811 webserver.cc:460] Webserver started at http://127.3.85.129:41361/ using document root <none> and password file <none>
I20260619 18:45:55.535987 22811 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:55.536164 22811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:55.536525 22811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:55.539880 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "febce28a14c54dfeb201f46675e410a8"
format_stamp: "Formatted at 2026-06-19 18:45:55 on dist-test-slave-09fp"
server_key: "bdb6d86a130c0fd5a08050d27b3ea733"
server_key_iv: "84ad21547d0d05e96cefedf6529ab439"
server_key_version: "encryptionkey@0"
I20260619 18:45:55.540822 22811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "febce28a14c54dfeb201f46675e410a8"
format_stamp: "Formatted at 2026-06-19 18:45:55 on dist-test-slave-09fp"
server_key: "bdb6d86a130c0fd5a08050d27b3ea733"
server_key_iv: "84ad21547d0d05e96cefedf6529ab439"
server_key_version: "encryptionkey@0"
I20260619 18:45:55.547129 22811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:45:55.551185 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:55.552826 22811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:45:55.553077 22811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "febce28a14c54dfeb201f46675e410a8"
format_stamp: "Formatted at 2026-06-19 18:45:55 on dist-test-slave-09fp"
server_key: "bdb6d86a130c0fd5a08050d27b3ea733"
server_key_iv: "84ad21547d0d05e96cefedf6529ab439"
server_key_version: "encryptionkey@0"
I20260619 18:45:55.553332 22811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:55.581013 22811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:55.585901 22811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:55.586261 22811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:55.587460 22811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:55.588934 22811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:55.589052 22811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:55.589215 22811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:55.589289 22811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:55.625054 22811 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:45921
I20260619 18:45:55.625126 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:45921 every 8 connection(s)
I20260619 18:45:55.626668 22811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:55.627887 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22811
I20260619 18:45:55.628087 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:55.628683 3414 external_mini_cluster.cc:1428] Setting key 979cf240392625ff8aaa7af851148d19
Jun 19 18:45:55 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894755, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.129@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:55.652478 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:55.629876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:59951 (local address 127.3.85.190:39937)
0619 18:45:55.630434 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:55.630442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:55.631744 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 18:45:55.634941 (+ 3197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:55.634946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:55.635065 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:55.635168 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:55.637542 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:55.638281 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:55.641168 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:55.641998 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:55.642211 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:55.646054 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:55.646060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:55.646064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:55.646102 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:55.648962 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:55.649872 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:55.649885 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:55.649888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:55.650020 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:55.650555 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:55.650559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:55.650562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:55.650851 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:55.651022 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:55.651817 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 18:45:55.652108 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":285,"threads_started":1}
I20260619 18:45:55.653551 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:55.630635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39937 (local address 127.3.85.129:59951)
0619 18:45:55.631527 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:55.631598 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:55.634563 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:55.635493 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:55.635509 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:55.636380 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:55.637309 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:55.637341 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:55.638532 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:55.638541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:55.640870 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:55.640902 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:55.642279 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:55.642285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:55.642601 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:55.643386 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:55.643412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:55.645754 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:55.649174 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:55.649184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:55.649215 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:55.649718 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:55.650155 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:55.650160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:55.650164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:55.650389 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:55.651002 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:55.651011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:55.651400 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 18:45:55.651691 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:45:55.652315 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":305,"threads_started":1}
I20260619 18:45:55.655470 22939 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39937
I20260619 18:45:55.655831 22939 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:55.656695 22939 heartbeater.cc:511] Master 127.3.85.190:39937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.130@KRBTEST.COM; defaulting to no policy
I20260619 18:45:55.659451 22750 ts_manager.cc:194] Registered new tserver with Master: febce28a14c54dfeb201f46675e410a8 (127.3.85.129:45921)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.130@KRBTEST.COM" created.
I20260619 18:45:55.661051 22750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.129@KRBTEST.COM'} at 127.3.85.129:59951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
Entry for principal kudu/127.3.85.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab.
I20260619 18:45:55.691051 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:56.039693 22945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:56.040004 22945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:56.040102 22945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:56.048056 22945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:56.048194 22945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:56.048316 22945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:56.059142 22945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.130.keytab
--principal=kudu/127.3.85.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:56.060508 22945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:56.061847 22945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:56.076375 22951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:56.076484 22950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:56.076392 22953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:56.077680 22945 server_base.cc:1034] running on GCE node
I20260619 18:45:56.078357 22945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:56.079464 22945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:56.080660 22945 hybrid_clock.cc:648] HybridClock initialized: now 1781894756080617 us; error 36 us; skew 500 ppm
I20260619 18:45:56.080993 22945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:56 dist-test-slave-09fp krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:56.084473 22945 init.cc:377] Logged in from keytab as kudu/127.3.85.130@KRBTEST.COM (short username kudu)
I20260619 18:45:56.086205 22945 webserver.cc:460] Webserver started at http://127.3.85.130:34921/ using document root <none> and password file <none>
I20260619 18:45:56.087055 22945 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:56.087213 22945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:56.087551 22945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:56.090806 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "0c1ca371de52485d9066b21af992ae6c"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "4dd5132f88d1267431f82c6adf4e00e7"
server_key_iv: "a0a14a5aa43b37eae5a5cef49ae715ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.091707 22945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "0c1ca371de52485d9066b21af992ae6c"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "4dd5132f88d1267431f82c6adf4e00e7"
server_key_iv: "a0a14a5aa43b37eae5a5cef49ae715ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.097672 22945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260619 18:45:56.101799 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.103521 22945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:45:56.103752 22945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "0c1ca371de52485d9066b21af992ae6c"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "4dd5132f88d1267431f82c6adf4e00e7"
server_key_iv: "a0a14a5aa43b37eae5a5cef49ae715ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.104003 22945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:56.121829 22945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:56.127085 22945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:56.127400 22945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:56.128458 22945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:56.129990 22945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:56.130120 22945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.130275 22945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:56.130353 22945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.165380 22945 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:37409
I20260619 18:45:56.165426 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:37409 every 8 connection(s)
I20260619 18:45:56.166836 22945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:56.168385 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 22945
I20260619 18:45:56.168579 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:56.169186 3414 external_mini_cluster.cc:1428] Setting key 67ff3905a2fb0c5e1bd20640f5642acd
Jun 19 18:45:56 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.130@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:56.194605 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.169978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:44045 (local address 127.3.85.190:39937)
0619 18:45:56.170557 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.170561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.171836 (+ 1275us) server_negotiation.cc:407] Connection header received
0619 18:45:56.175034 (+ 3198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.175039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.175115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.175208 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:56.177674 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.178514 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.181243 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.181695 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.181889 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.186221 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:56.186231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:56.186239 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:56.186294 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:56.190650 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.191634 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.191642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.191649 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.191842 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.192350 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.192359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.192365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.192795 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:56.192991 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.193716 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.194020 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260619 18:45:56.195312 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.170628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39937 (local address 127.3.85.130:44045)
0619 18:45:56.171633 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.171692 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.174631 (+ 2939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.175547 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.175563 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.176473 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:56.177420 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.177450 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.178721 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.178729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.181038 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.181061 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.181988 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.181994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.182229 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.183148 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:56.183187 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:56.185852 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:56.190873 (+ 5021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:56.190883 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:56.190901 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:56.191445 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:56.191978 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:56.191982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:56.191984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:56.192216 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:56.192931 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:56.192938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:56.193312 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.193578 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.194021 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":448,"threads_started":1}
I20260619 18:45:56.197067 23073 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39937
I20260619 18:45:56.197403 23073 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:56.198216 23073 heartbeater.cc:511] Master 127.3.85.190:39937 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.85.131@KRBTEST.COM; defaulting to no policy
I20260619 18:45:56.200342 22750 ts_manager.cc:194] Registered new tserver with Master: 0c1ca371de52485d9066b21af992ae6c (127.3.85.130:37409)
I20260619 18:45:56.201651 22750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.130@KRBTEST.COM'} at 127.3.85.130:44045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
Entry for principal kudu/127.3.85.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab.
I20260619 18:45:56.234045 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:45:56.581290 23079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:56.581612 23079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:56.581714 23079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:56.589360 23079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:56.589494 23079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:56.589617 23079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:56.600051 23079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.131.keytab
--principal=kudu/127.3.85.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:39937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:56.601395 23079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:56.602748 23079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:56.616824 23084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:56.616861 23087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:56.616794 23085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:56.617930 23079 server_base.cc:1034] running on GCE node
I20260619 18:45:56.618753 23079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:56.619825 23079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:56.621084 23079 hybrid_clock.cc:648] HybridClock initialized: now 1781894756621034 us; error 45 us; skew 500 ppm
I20260619 18:45:56.621485 23079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:45:56 dist-test-slave-09fp krb5kdc[22705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:45:56.624969 23079 init.cc:377] Logged in from keytab as kudu/127.3.85.131@KRBTEST.COM (short username kudu)
I20260619 18:45:56.626963 23079 webserver.cc:460] Webserver started at http://127.3.85.131:46251/ using document root <none> and password file <none>
I20260619 18:45:56.627717 23079 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:56.627853 23079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:56.628198 23079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:56.631326 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "5f4c61b8a87a48e1a7464f37d0b30521"
server_key_iv: "091988cad22b31c398cc2a1347e872dc"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.632212 23079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "5f4c61b8a87a48e1a7464f37d0b30521"
server_key_iv: "091988cad22b31c398cc2a1347e872dc"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.637774 23079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:45:56.641582 23094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.643167 23079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:56.643362 23079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1"
format_stamp: "Formatted at 2026-06-19 18:45:56 on dist-test-slave-09fp"
server_key: "5f4c61b8a87a48e1a7464f37d0b30521"
server_key_iv: "091988cad22b31c398cc2a1347e872dc"
server_key_version: "encryptionkey@0"
I20260619 18:45:56.643584 23079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:56.664441 22939 heartbeater.cc:503] Master 127.3.85.190:39937 was elected leader, sending a full tablet report...
I20260619 18:45:56.676991 23079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:56.681229 23079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:56.681486 23079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:56.682502 23079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:56.683981 23079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:56.684059 23079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.684154 23079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:56.684196 23079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:56.724177 23079 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:38147
I20260619 18:45:56.724220 23206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:38147 every 8 connection(s)
I20260619 18:45:56.725559 23079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:56.730918 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23079
I20260619 18:45:56.731130 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:56.731611 3414 external_mini_cluster.cc:1428] Setting key 75664b92825062cb8d6c651dfa992f0b
Jun 19 18:45:56 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894756, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.131@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:45:56.746665 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.728302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:36805 (local address 127.3.85.190:39937)
0619 18:45:56.728791 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.728796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.729736 (+ 940us) server_negotiation.cc:407] Connection header received
0619 18:45:56.731841 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.731847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.731943 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.732087 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:56.733662 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.734576 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.736179 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.736593 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.736745 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.740537 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:56.740542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:56.740545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:56.740581 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:56.743262 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.744069 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.744079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.744082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.744224 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.744688 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.744695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.744697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.744989 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:56.745162 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.745923 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.746184 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260619 18:45:56.747773 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.728816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39937 (local address 127.3.85.131:36805)
0619 18:45:56.729568 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.729608 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.731513 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.732341 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.732351 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.732881 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:56.733483 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.733502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.734740 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.734744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.736016 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.736030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.736852 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.736859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.737121 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.737863 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:45:56.737888 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:56.740280 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:56.743437 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:56.743447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:56.743467 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:56.743931 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:56.744347 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:45:56.744352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:45:56.744355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:45:56.744571 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:45:56.745128 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:56.745137 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:56.745523 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.745826 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.746404 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
I20260619 18:45:56.749190 23207 heartbeater.cc:347] Connected to a master server at 127.3.85.190:39937
I20260619 18:45:56.749531 23207 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:56.750301 23207 heartbeater.cc:511] Master 127.3.85.190:39937 requested a full tablet report, sending...
I20260619 18:45:56.751986 22750 ts_manager.cc:194] Registered new tserver with Master: 5a17f03e981e4fdba4ea7dbfaac83fb1 (127.3.85.131:38147)
I20260619 18:45:56.752799 22750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.85.131@KRBTEST.COM'} at 127.3.85.131:36805
I20260619 18:45:56.759953 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:56.777297 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.764317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.3.85.190:39937)
0619 18:45:56.764490 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.764495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.764967 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:45:56.765116 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.765119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.765190 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.765282 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:56.766768 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.767525 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.768767 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.769176 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.769276 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.771826 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:56.771831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:56.771835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:56.771873 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:56.774755 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.775415 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.775419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.775421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.775546 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:56.775962 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:56.775965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:56.775967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:56.776311 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:56.776506 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.776845 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.777048 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:45:56.781651 22750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:56.789059 22750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:56.815222 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.801885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:37409 (local address 127.0.0.1:58772)
0619 18:45:56.803188 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.803234 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.803405 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.805733 (+ 2328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.805743 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.805776 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:56.806618 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.806633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.808758 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.808764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.811573 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.811591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.814037 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.814045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.814348 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.814387 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.814519 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.814737 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":413,"thread_start_us":2508,"threads_started":1}
I20260619 18:45:56.815332 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.801927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.3.85.130:37409)
0619 18:45:56.804002 (+ 2075us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.804014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.804052 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:45:56.804164 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.804172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.804447 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.805886 (+ 1439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:56.806877 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.808605 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.811818 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.813482 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.813639 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.813876 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.814640 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.814852 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1845,"thread_start_us":1721,"threads_started":1}
I20260619 18:45:56.815246 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.802844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.3.85.131:38147)
0619 18:45:56.804499 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.804507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.804840 (+ 333us) server_negotiation.cc:407] Connection header received
0619 18:45:56.805043 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.805051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.805267 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.805472 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:56.807207 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.808886 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.812233 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.813513 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.813662 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.813906 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.814679 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.814889 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":1230,"threads_started":1}
I20260619 18:45:56.816139 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.802163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:38147 (local address 127.0.0.1:39076)
0619 18:45:56.804704 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.804722 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.804844 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.805763 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.805766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.805782 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:56.806968 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.806986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.809069 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.809074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.812058 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.812071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.814077 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.814085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.814316 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.814359 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.815564 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.815718 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2392,"mutex_wait_us":73,"thread_start_us":959,"threads_started":1}
I20260619 18:45:56.817034 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.805033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:45921 (local address 127.0.0.1:54002)
0619 18:45:56.807157 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.807175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.807312 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.807930 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.807980 (+ 50us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.808009 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:56.808369 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.808382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.810629 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.810634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.812882 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.812898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.815182 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.815190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.815395 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.815430 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.816541 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.816691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"thread_start_us":1814,"threads_started":1}
I20260619 18:45:56.817873 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.805199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.3.85.129:45921)
0619 18:45:56.806722 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.806733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.807298 (+ 565us) server_negotiation.cc:407] Connection header received
0619 18:45:56.807472 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.807480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.807728 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.807916 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:56.808552 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.810430 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.813093 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.814658 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.814789 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.815023 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.817287 (+ 2264us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.817491 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1125,"threads_started":1}
I20260619 18:45:56.819895 23142 tablet_service.cc:1467] Processing CreateTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 (DEFAULT_TABLE table=test-table [id=ca5be2b70d494f8683aee8340fa3cadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:56.819913 23008 tablet_service.cc:1467] Processing CreateTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 (DEFAULT_TABLE table=test-table [id=ca5be2b70d494f8683aee8340fa3cadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:56.821308 23142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1288c984a1514bf79a3c89fbe85087a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:56.821375 23008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1288c984a1514bf79a3c89fbe85087a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:56.822261 22874 tablet_service.cc:1467] Processing CreateTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 (DEFAULT_TABLE table=test-table [id=ca5be2b70d494f8683aee8340fa3cadf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:56.823933 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1288c984a1514bf79a3c89fbe85087a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:56.834380 23226 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Bootstrap starting.
I20260619 18:45:56.837131 23227 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Bootstrap starting.
I20260619 18:45:56.838574 23226 tablet_bootstrap.cc:654] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:56.839689 23228 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Bootstrap starting.
I20260619 18:45:56.840010 23226 log.cc:824] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:56.841287 23227 tablet_bootstrap.cc:654] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:56.842641 23227 log.cc:824] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:56.843900 23228 tablet_bootstrap.cc:654] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:56.844122 23226 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: No bootstrap required, opened a new log
I20260619 18:45:56.844424 23226 ts_tablet_manager.cc:1397] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260619 18:45:56.845041 23228 log.cc:824] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:56.846000 23227 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: No bootstrap required, opened a new log
I20260619 18:45:56.846294 23227 ts_tablet_manager.cc:1397] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 18:45:56.848021 23228 tablet_bootstrap.cc:492] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: No bootstrap required, opened a new log
I20260619 18:45:56.848342 23228 ts_tablet_manager.cc:1397] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 18:45:56.856799 23226 raft_consensus.cc:348] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.857385 23226 raft_consensus.cc:374] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:56.857510 23226 raft_consensus.cc:729] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a17f03e981e4fdba4ea7dbfaac83fb1, State: Initialized, Role: FOLLOWER
I20260619 18:45:56.857604 23227 raft_consensus.cc:348] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.858088 23227 raft_consensus.cc:374] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:56.858183 23227 raft_consensus.cc:729] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1ca371de52485d9066b21af992ae6c, State: Initialized, Role: FOLLOWER
I20260619 18:45:56.858379 23226 consensus_queue.cc:260] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.858011 23228 raft_consensus.cc:348] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.858893 23228 raft_consensus.cc:374] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:56.858985 23228 raft_consensus.cc:729] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febce28a14c54dfeb201f46675e410a8, State: Initialized, Role: FOLLOWER
I20260619 18:45:56.858829 23227 consensus_queue.cc:260] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.859938 23207 heartbeater.cc:503] Master 127.3.85.190:39937 was elected leader, sending a full tablet report...
I20260619 18:45:56.860514 23073 heartbeater.cc:503] Master 127.3.85.190:39937 was elected leader, sending a full tablet report...
I20260619 18:45:56.860459 23228 consensus_queue.cc:260] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.861181 23227 ts_tablet_manager.cc:1428] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260619 18:45:56.862273 23226 ts_tablet_manager.cc:1428] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260619 18:45:56.863022 23228 ts_tablet_manager.cc:1428] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260619 18:45:56.879513 22940 tablet.cc:2369] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:56.919121 23074 tablet.cc:2369] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:56.953047 23234 raft_consensus.cc:482] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:45:56.953372 23234 raft_consensus.cc:504] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.955514 23234 leader_election.cc:288] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1ca371de52485d9066b21af992ae6c (127.3.85.130:37409), 5a17f03e981e4fdba4ea7dbfaac83fb1 (127.3.85.131:38147)
I20260619 18:45:56.964164 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.956515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:37007 (local address 127.3.85.130:37409)
0619 18:45:56.957015 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.957022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.957874 (+ 852us) server_negotiation.cc:407] Connection header received
0619 18:45:56.958129 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.958135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.958226 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.958307 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:56.959271 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.960129 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.962170 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.963151 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.963348 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.963428 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.963750 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.963902 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260619 18:45:56.964344 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.955706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:37409 (local address 127.3.85.129:37007)
0619 18:45:56.957718 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.957744 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.957947 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.958560 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.958566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.958592 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:56.959064 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.959078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.960291 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.960295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.961982 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.961995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.963370 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.963376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.963562 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.963599 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.963694 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.963840 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1829,"mutex_wait_us":4,"thread_start_us":1933,"threads_started":1}
I20260619 18:45:56.965539 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.956122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:51155 (local address 127.3.85.131:38147)
0619 18:45:56.956810 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:56.956819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:56.958065 (+ 1246us) server_negotiation.cc:407] Connection header received
0619 18:45:56.958354 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:56.958360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:56.958434 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:56.958535 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:56.959450 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.960549 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.963274 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.964462 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.964616 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.964695 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:56.965118 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 18:45:56.965280 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
I20260619 18:45:56.965583 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:56.955849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:38147 (local address 127.3.85.129:51155)
0619 18:45:56.957945 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:56.957964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:56.958180 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:56.958779 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:56.958783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:56.958813 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:56.959274 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.959285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.960748 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.960753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.963089 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:56.963105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:56.964727 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:56.964733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:56.964926 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:56.964960 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:45:56.965054 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:45:56.965193 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969,"mutex_wait_us":103,"thread_start_us":1385,"threads_started":1}
I20260619 18:45:56.965703 23028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1288c984a1514bf79a3c89fbe85087a3" candidate_uuid: "febce28a14c54dfeb201f46675e410a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1ca371de52485d9066b21af992ae6c" is_pre_election: true
I20260619 18:45:56.966274 23028 raft_consensus.cc:2454] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febce28a14c54dfeb201f46675e410a8 in term 0.
I20260619 18:45:56.966958 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1288c984a1514bf79a3c89fbe85087a3" candidate_uuid: "febce28a14c54dfeb201f46675e410a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" is_pre_election: true
I20260619 18:45:56.967140 22827 leader_election.cc:302] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1ca371de52485d9066b21af992ae6c, febce28a14c54dfeb201f46675e410a8; no voters:
I20260619 18:45:56.967733 23162 raft_consensus.cc:2454] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febce28a14c54dfeb201f46675e410a8 in term 0.
I20260619 18:45:56.967821 23234 raft_consensus.cc:2781] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:45:56.967975 23234 raft_consensus.cc:482] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:45:56.968065 23234 raft_consensus.cc:3037] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:56.971375 23234 raft_consensus.cc:504] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.972172 23234 leader_election.cc:288] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1ca371de52485d9066b21af992ae6c (127.3.85.130:37409), 5a17f03e981e4fdba4ea7dbfaac83fb1 (127.3.85.131:38147)
I20260619 18:45:56.972828 23162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1288c984a1514bf79a3c89fbe85087a3" candidate_uuid: "febce28a14c54dfeb201f46675e410a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1"
I20260619 18:45:56.972836 23028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1288c984a1514bf79a3c89fbe85087a3" candidate_uuid: "febce28a14c54dfeb201f46675e410a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1ca371de52485d9066b21af992ae6c"
I20260619 18:45:56.973093 23162 raft_consensus.cc:3037] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:56.973083 23028 raft_consensus.cc:3037] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:56.975554 23162 raft_consensus.cc:2454] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febce28a14c54dfeb201f46675e410a8 in term 1.
I20260619 18:45:56.976140 22828 leader_election.cc:302] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a17f03e981e4fdba4ea7dbfaac83fb1, febce28a14c54dfeb201f46675e410a8; no voters:
I20260619 18:45:56.976409 23028 raft_consensus.cc:2454] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febce28a14c54dfeb201f46675e410a8 in term 1.
I20260619 18:45:56.976516 23234 raft_consensus.cc:2781] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:56.977138 23234 raft_consensus.cc:686] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 1 LEADER]: Becoming Leader. State: Replica: febce28a14c54dfeb201f46675e410a8, State: Running, Role: LEADER
W20260619 18:45:56.977346 23208 tablet.cc:2369] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:45:56.977771 23234 consensus_queue.cc:237] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } }
I20260619 18:45:56.983352 22748 catalog_manager.cc:5495] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to febce28a14c54dfeb201f46675e410a8 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "febce28a14c54dfeb201f46675e410a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "febce28a14c54dfeb201f46675e410a8" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 45921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 } health_report { overall_health: UNKNOWN } } }
I20260619 18:45:57.009734 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:57.004076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.3.85.129:45921)
0619 18:45:57.004276 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:57.004281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:57.004300 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:45:57.004394 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:57.004397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:57.004489 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:57.004624 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:45:57.005066 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.005790 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.007374 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.007804 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.007911 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:57.008418 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:45:57.009034 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:45:57.009211 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:57.009336 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 18:45:57.009411 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:45:57.018554 23028 raft_consensus.cc:1261] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 1 FOLLOWER]: Refusing update from remote peer febce28a14c54dfeb201f46675e410a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:57.018558 23162 raft_consensus.cc:1261] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 1 FOLLOWER]: Refusing update from remote peer febce28a14c54dfeb201f46675e410a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:45:57.019663 23237 consensus_queue.cc:1035] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1ca371de52485d9066b21af992ae6c" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:57.020063 23234 consensus_queue.cc:1035] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a17f03e981e4fdba4ea7dbfaac83fb1" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:45:57.051716 22748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:57.052052 22748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:45:57.056809 22748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 161e82bd06774c96858e757d9129d83b: Sending DeleteTablet for 3 replicas of tablet 1288c984a1514bf79a3c89fbe85087a3
I20260619 18:45:57.058513 23008 tablet_service.cc:1514] Processing DeleteTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:57 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:58772
I20260619 18:45:57.058513 23142 tablet_service.cc:1514] Processing DeleteTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:57 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:39076
I20260619 18:45:57.058571 22874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1288c984a1514bf79a3c89fbe85087a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:45:57 UTC) from {username='kudu', principal='kudu/127.3.85.190@KRBTEST.COM'} at 127.0.0.1:54002
I20260619 18:45:57.060822 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22811
I20260619 18:45:57.069931 23251 tablet_replica.cc:330] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: stopping tablet replica
I20260619 18:45:57.070508 23251 raft_consensus.cc:2229] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:57.071130 23251 raft_consensus.cc:2258] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:57.072052 23252 tablet_replica.cc:330] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: stopping tablet replica
I20260619 18:45:57.072547 23252 raft_consensus.cc:2229] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:45:57.073284 23252 raft_consensus.cc:2258] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:57.073772 23250 tablet_replica.cc:330] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: stopping tablet replica
I20260619 18:45:57.074229 23250 raft_consensus.cc:2229] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:45:57.074806 23250 raft_consensus.cc:2258] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:45:57.076086 23252 ts_tablet_manager.cc:1905] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:57.076093 23251 ts_tablet_manager.cc:1905] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:57.076916 23250 ts_tablet_manager.cc:1905] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:45:57.083559 23251 ts_tablet_manager.cc:1918] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:57.083770 23251 log.cc:1196] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/1288c984a1514bf79a3c89fbe85087a3
I20260619 18:45:57.083901 23252 ts_tablet_manager.cc:1918] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:57.084170 23252 log.cc:1196] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-0/wal/wals/1288c984a1514bf79a3c89fbe85087a3
I20260619 18:45:57.084324 23251 ts_tablet_manager.cc:1939] T 1288c984a1514bf79a3c89fbe85087a3 P 5a17f03e981e4fdba4ea7dbfaac83fb1: Deleting consensus metadata
I20260619 18:45:57.084367 23250 ts_tablet_manager.cc:1918] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:45:57.084538 23250 log.cc:1196] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/1288c984a1514bf79a3c89fbe85087a3
I20260619 18:45:57.084986 23252 ts_tablet_manager.cc:1939] T 1288c984a1514bf79a3c89fbe85087a3 P febce28a14c54dfeb201f46675e410a8: Deleting consensus metadata
I20260619 18:45:57.085078 23250 ts_tablet_manager.cc:1939] T 1288c984a1514bf79a3c89fbe85087a3 P 0c1ca371de52485d9066b21af992ae6c: Deleting consensus metadata
I20260619 18:45:57.086508 22735 catalog_manager.cc:4841] TS 5a17f03e981e4fdba4ea7dbfaac83fb1 (127.3.85.131:38147): tablet 1288c984a1514bf79a3c89fbe85087a3 (table test-table [id=ca5be2b70d494f8683aee8340fa3cadf]) successfully deleted
I20260619 18:45:57.086833 22734 catalog_manager.cc:4841] TS 0c1ca371de52485d9066b21af992ae6c (127.3.85.130:37409): tablet 1288c984a1514bf79a3c89fbe85087a3 (table test-table [id=ca5be2b70d494f8683aee8340fa3cadf]) successfully deleted
I20260619 18:45:57.087772 22734 catalog_manager.cc:4841] TS febce28a14c54dfeb201f46675e410a8 (127.3.85.129:45921): tablet 1288c984a1514bf79a3c89fbe85087a3 (table test-table [id=ca5be2b70d494f8683aee8340fa3cadf]) successfully deleted
Jun 19 18:45:57 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.129@KRBTEST.COM
I20260619 18:45:57.101454 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:57.061764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.3.85.129:45921)
0619 18:45:57.062053 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:57.062062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:57.077267 (+ 15205us) server_negotiation.cc:407] Connection header received
0619 18:45:57.077748 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:57.077753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:57.077874 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:57.077969 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:57.079833 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.081056 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.087136 (+ 6080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.087939 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.088069 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:57.093311 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:57.093327 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:57.093347 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:57.093403 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:57.097632 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.098904 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.098916 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.098921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.099131 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.099609 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.099615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.099619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.100111 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:57.100472 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:57.100775 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:45:57.101053 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:45:57.102294 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:57.202327 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22811
I20260619 18:45:57.223377 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22945
Jun 19 18:45:57 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.130@KRBTEST.COM
I20260619 18:45:57.239888 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:57.224218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.3.85.130:37409)
0619 18:45:57.224423 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:57.224428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:57.224448 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:45:57.224556 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:57.224560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:57.224627 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:57.224724 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:57.226163 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.227060 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.228490 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.228884 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.228987 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:57.233461 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:57.233481 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:57.233498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:57.233545 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:57.236708 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.237572 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.237581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.237583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.237742 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.238110 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.238113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.238116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.238674 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:57.238931 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:57.239279 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 18:45:57.239472 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:45:57.240769 23068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:57.332708 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22945
I20260619 18:45:57.350651 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23079
Jun 19 18:45:57 dist-test-slave-09fp krb5kdc[22705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.131@KRBTEST.COM
I20260619 18:45:57.367440 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:57.351453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.3.85.131:38147)
0619 18:45:57.351665 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:57.351671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:57.351694 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:45:57.351786 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:57.351791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:57.351864 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:57.351984 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:57.353394 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.354152 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.355763 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:57.356199 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:57.356319 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:57.360976 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:57.360993 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:57.361010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:45:57.361054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:57.364149 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.365078 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.365088 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.365090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.365243 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:45:57.365646 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:45:57.365650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:45:57.365652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:45:57.366039 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:57.366277 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:57.366833 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 18:45:57.367037 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:45:57.368219 23202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:57.459648 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23079
I20260619 18:45:57.476538 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 22718
I20260619 18:45:57.477662 22780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:45:57.591027 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 22718
2026-06-19T18:45:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4532 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T18:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:45:57Z Disabled control of system clock
I20260619 18:45:57.672344 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:36987
--webserver_interface=127.3.85.190
--webserver_port=0
--builtin_ntp_servers=127.3.85.148:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:58.024041 23261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:58.024360 23261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:58.024446 23261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:58.032083 23261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:45:58.032213 23261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:58.032253 23261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:58.032302 23261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:45:58.032348 23261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:45:58.042030 23261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:36987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:58.043354 23261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:58.045192 23261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:58.058094 23269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:58.058215 23267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:58.058192 23266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:58.059571 23261 server_base.cc:1034] running on GCE node
I20260619 18:45:58.060478 23261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:58.062049 23261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:58.063297 23261 hybrid_clock.cc:648] HybridClock initialized: now 1781894758063224 us; error 73 us; skew 500 ppm
I20260619 18:45:58.063687 23261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:58.066265 23261 webserver.cc:460] Webserver started at http://127.3.85.190:41551/ using document root <none> and password file <none>
I20260619 18:45:58.067118 23261 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:58.067269 23261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:58.067642 23261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:58.070959 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "6f3e85861e774697a1e044539871eb90"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "73217c1e4e552b72c2f9fdc280f8a9cb"
server_key_iv: "e9838cb44ed8973deb50094ad48f2f8e"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.071916 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "6f3e85861e774697a1e044539871eb90"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "73217c1e4e552b72c2f9fdc280f8a9cb"
server_key_iv: "e9838cb44ed8973deb50094ad48f2f8e"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.077855 23261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:45:58.081918 23275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:58.083590 23261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:45:58.083835 23261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "6f3e85861e774697a1e044539871eb90"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "73217c1e4e552b72c2f9fdc280f8a9cb"
server_key_iv: "e9838cb44ed8973deb50094ad48f2f8e"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.084108 23261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:58.109732 23261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:58.110833 23261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:58.111158 23261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:58.131358 23261 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:36987
I20260619 18:45:58.131379 23326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:36987 every 8 connection(s)
I20260619 18:45:58.132826 23261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:45:58.136935 23327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:58.139276 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23261
I20260619 18:45:58.139482 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:45:58.140044 3414 external_mini_cluster.cc:1428] Setting key 590b5634647f0158e8d3d7e8aad283e1
I20260619 18:45:58.150290 23327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: Bootstrap starting.
I20260619 18:45:58.153962 23327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:58.155078 23327 log.cc:824] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:58.158226 23327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: No bootstrap required, opened a new log
I20260619 18:45:58.160113 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:58.142301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.3.85.190:36987)
0619 18:45:58.143582 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:58.143594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:58.143648 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:45:58.145515 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:58.145542 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:58.146063 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:58.146597 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:58.147321 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.148681 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.150647 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.151347 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.151483 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:58.157547 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:58.157556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:58.157572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:58.158158 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:58.158250 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:58.158276 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:58.158446 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:58.158768 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 18:45:58.159106 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":458,"threads_started":1}
I20260619 18:45:58.167472 23327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } }
I20260619 18:45:58.167918 23327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:58.168041 23327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3e85861e774697a1e044539871eb90, State: Initialized, Role: FOLLOWER
I20260619 18:45:58.168649 23327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } }
I20260619 18:45:58.168910 23327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:45:58.169071 23327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:45:58.169252 23327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:45:58.171640 23327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } }
I20260619 18:45:58.172290 23327 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3e85861e774697a1e044539871eb90; no voters:
I20260619 18:45:58.173039 23327 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:45:58.173206 23332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:45:58.174099 23332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3e85861e774697a1e044539871eb90, State: Running, Role: LEADER
I20260619 18:45:58.174777 23332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } }
I20260619 18:45:58.175653 23327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:45:58.177896 23334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3e85861e774697a1e044539871eb90. Latest consensus state: current_term: 1 leader_uuid: "6f3e85861e774697a1e044539871eb90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } } }
I20260619 18:45:58.178371 23334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:58.177874 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3e85861e774697a1e044539871eb90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3e85861e774697a1e044539871eb90" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 36987 } } }
I20260619 18:45:58.178614 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90 [sys.catalog]: This master's current role is: LEADER
I20260619 18:45:58.181532 23338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:45:58.187667 23338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:45:58.199095 23338 catalog_manager.cc:1347] Generated new cluster ID: c5bef5b4475c45498d5b5c1f9cfcbb6b
I20260619 18:45:58.199227 23338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:45:58.227460 23338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:45:58.228888 23338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:45:58.242774 23338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: Generated new TSK 0
I20260619 18:45:58.243773 23338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:45:58.250658 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.129:0
--local_ip_for_outbound_sockets=127.3.85.129
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--builtin_ntp_servers=127.3.85.148:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:45:58.612929 23351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:58.613252 23351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:58.613344 23351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:58.621235 23351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:58.621385 23351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:58.621503 23351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.129
I20260619 18:45:58.631186 23351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.85.129
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.129
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:58.632539 23351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:58.634410 23351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:58.648761 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:58.648720 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:58.648787 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:58.649907 23351 server_base.cc:1034] running on GCE node
I20260619 18:45:58.650789 23351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:58.651924 23351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:58.653175 23351 hybrid_clock.cc:648] HybridClock initialized: now 1781894758653116 us; error 62 us; skew 500 ppm
I20260619 18:45:58.653541 23351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:58.656111 23351 webserver.cc:460] Webserver started at http://127.3.85.129:42087/ using document root <none> and password file <none>
I20260619 18:45:58.656992 23351 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:58.657145 23351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:58.657517 23351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:58.661551 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data/instance:
uuid: "6f6f3a27d7e74641b32a09326c61bbd2"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "06a5ae6db796abca1c925ebb018a80b6"
server_key_iv: "ae39423715b7949b333747d6711314ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.662551 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6f3a27d7e74641b32a09326c61bbd2"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "06a5ae6db796abca1c925ebb018a80b6"
server_key_iv: "ae39423715b7949b333747d6711314ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.668371 23351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:45:58.672416 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:58.673889 23351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:45:58.674086 23351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal
uuid: "6f6f3a27d7e74641b32a09326c61bbd2"
format_stamp: "Formatted at 2026-06-19 18:45:58 on dist-test-slave-09fp"
server_key: "06a5ae6db796abca1c925ebb018a80b6"
server_key_iv: "ae39423715b7949b333747d6711314ec"
server_key_version: "encryptionkey@0"
I20260619 18:45:58.674346 23351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:58.690321 23351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:58.691255 23351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:58.691586 23351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:58.692628 23351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:58.694252 23351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:58.694404 23351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:58.694566 23351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:58.694658 23351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:58.735368 23351 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.129:39695
I20260619 18:45:58.735469 23477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.129:39695 every 8 connection(s)
I20260619 18:45:58.736752 23351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/data/info.pb
I20260619 18:45:58.742334 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23351
I20260619 18:45:58.742553 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-0/wal/instance
I20260619 18:45:58.743023 3414 external_mini_cluster.cc:1428] Setting key 2c8f84479dbc81e036b874912ba0aa9c
I20260619 18:45:58.746178 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.130:0
--local_ip_for_outbound_sockets=127.3.85.130
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--builtin_ntp_servers=127.3.85.148:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:58.752317 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:58.739506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:55487 (local address 127.3.85.190:36987)
0619 18:45:58.739966 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:58.739972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:58.741232 (+ 1260us) server_negotiation.cc:407] Connection header received
0619 18:45:58.743726 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:58.743753 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:58.743853 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:58.744005 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:58.745543 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.746690 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.748543 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.749247 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.749389 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:58.751134 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:58.751143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:58.751152 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:58.751181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:58.751237 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:58.751257 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:58.751357 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:58.751703 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 18:45:58.751826 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260619 18:45:58.753875 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:58.740138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:36987 (local address 127.3.85.129:55487)
0619 18:45:58.741058 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:58.741100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:58.743371 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:58.744331 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:58.744343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:58.744477 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:58.745357 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:58.745380 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.747009 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.747014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:58.748351 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:58.748365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:58.749505 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:58.749510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:58.749727 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:58.750686 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:58.750690 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:58.750747 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:58.750868 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:58.751427 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:58.751440 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:58.751450 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:45:58.751663 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:45:58.752201 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":290,"threads_started":1}
I20260619 18:45:58.756201 23478 heartbeater.cc:347] Connected to a master server at 127.3.85.190:36987
I20260619 18:45:58.756593 23478 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:58.757582 23478 heartbeater.cc:511] Master 127.3.85.190:36987 requested a full tablet report, sending...
I20260619 18:45:58.760135 23292 ts_manager.cc:194] Registered new tserver with Master: 6f6f3a27d7e74641b32a09326c61bbd2 (127.3.85.129:39695)
I20260619 18:45:58.761714 23292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.129:55487
W20260619 18:45:59.106381 23482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:59.106829 23482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:59.106985 23482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:59.114732 23482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:59.114894 23482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:59.115078 23482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.130
I20260619 18:45:59.124958 23482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.85.130
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.130
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:59.126399 23482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:59.128330 23482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:59.143031 23487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:59.145195 23488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:59.147032 23490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:59.147203 23482 server_base.cc:1034] running on GCE node
I20260619 18:45:59.148022 23482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:59.149287 23482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:59.150571 23482 hybrid_clock.cc:648] HybridClock initialized: now 1781894759150528 us; error 61 us; skew 500 ppm
I20260619 18:45:59.150964 23482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:59.154057 23482 webserver.cc:460] Webserver started at http://127.3.85.130:37141/ using document root <none> and password file <none>
I20260619 18:45:59.155021 23482 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:59.155149 23482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:59.155501 23482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:59.158815 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data/instance:
uuid: "0202aa3a337f45edb478e1c4033676c0"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "bfa7b72c92c6ef7d11c8506334624804"
server_key_iv: "db7eb652515942c75d4f7e61cc0b5c4b"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.159626 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance:
uuid: "0202aa3a337f45edb478e1c4033676c0"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "bfa7b72c92c6ef7d11c8506334624804"
server_key_iv: "db7eb652515942c75d4f7e61cc0b5c4b"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.165251 23482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:45:59.169149 23496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.170904 23482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:45:59.171128 23482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal
uuid: "0202aa3a337f45edb478e1c4033676c0"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "bfa7b72c92c6ef7d11c8506334624804"
server_key_iv: "db7eb652515942c75d4f7e61cc0b5c4b"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.171391 23482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:59.190135 23482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:59.191082 23482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:59.191403 23482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:59.192440 23482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:59.193928 23482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:59.194054 23482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.194207 23482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:59.194298 23482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.228624 23482 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.130:45325
I20260619 18:45:59.228681 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.130:45325 every 8 connection(s)
I20260619 18:45:59.230046 23482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/data/info.pb
I20260619 18:45:59.237514 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23482
I20260619 18:45:59.237661 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal/instance
I20260619 18:45:59.238232 3414 external_mini_cluster.cc:1428] Setting key 958d9d06b8ecc5573be27a491e48622e
I20260619 18:45:59.241497 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.85.131:0
--local_ip_for_outbound_sockets=127.3.85.131
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--builtin_ntp_servers=127.3.85.148:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:45:59.244689 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.232934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.130:58075 (local address 127.3.85.190:36987)
0619 18:45:59.233163 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.233170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.234696 (+ 1526us) server_negotiation.cc:407] Connection header received
0619 18:45:59.236566 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.236571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.236646 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.236746 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:59.238045 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.239135 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.241377 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.242083 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.242245 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.243814 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:59.243819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:59.243822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:59.243860 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:59.243893 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:59.243905 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:59.244013 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.244264 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.244375 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:45:59.245837 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.233397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:36987 (local address 127.3.85.130:58075)
0619 18:45:59.234516 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:59.234566 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:59.236272 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:59.236981 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:59.236990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:59.237086 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:59.237852 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.237872 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.239318 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.239323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.241180 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.241196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.242318 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.242323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.242620 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.243436 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:59.243439 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:59.243493 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:59.243592 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:59.244029 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:59.244047 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:59.244055 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:45:59.244248 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 18:45:59.244716 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":334,"threads_started":1}
I20260619 18:45:59.247452 23609 heartbeater.cc:347] Connected to a master server at 127.3.85.190:36987
I20260619 18:45:59.247824 23609 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:59.248832 23609 heartbeater.cc:511] Master 127.3.85.190:36987 requested a full tablet report, sending...
I20260619 18:45:59.250814 23292 ts_manager.cc:194] Registered new tserver with Master: 0202aa3a337f45edb478e1c4033676c0 (127.3.85.130:45325)
I20260619 18:45:59.251690 23292 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.130:58075
W20260619 18:45:59.590095 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:45:59.590454 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:45:59.590674 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:45:59.598290 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:45:59.598469 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:45:59.598730 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.85.131
I20260619 18:45:59.608305 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.85.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.85.131
--webserver_port=0
--tserver_master_addrs=127.3.85.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.85.131
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:45:59.609716 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:45:59.611624 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:45:59.625738 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:59.625769 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:45:59.625705 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:45:59.627338 23612 server_base.cc:1034] running on GCE node
I20260619 18:45:59.628098 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:45:59.629271 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:45:59.630575 23612 hybrid_clock.cc:648] HybridClock initialized: now 1781894759630503 us; error 66 us; skew 500 ppm
I20260619 18:45:59.630951 23612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:45:59.633458 23612 webserver.cc:460] Webserver started at http://127.3.85.131:44263/ using document root <none> and password file <none>
I20260619 18:45:59.634231 23612 fs_manager.cc:359] Metadata directory not provided
I20260619 18:45:59.634328 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:45:59.634732 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:45:59.637672 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data/instance:
uuid: "c96a388eb0634ae58df903191d9f4ecf"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "58e2b1f42c38ebf91567bba3de6eed00"
server_key_iv: "767cfd385773c484a355c7a772ba3103"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.638527 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance:
uuid: "c96a388eb0634ae58df903191d9f4ecf"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "58e2b1f42c38ebf91567bba3de6eed00"
server_key_iv: "767cfd385773c484a355c7a772ba3103"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.644250 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:45:59.648063 23626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.649601 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:45:59.649833 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal
uuid: "c96a388eb0634ae58df903191d9f4ecf"
format_stamp: "Formatted at 2026-06-19 18:45:59 on dist-test-slave-09fp"
server_key: "58e2b1f42c38ebf91567bba3de6eed00"
server_key_iv: "767cfd385773c484a355c7a772ba3103"
server_key_version: "encryptionkey@0"
I20260619 18:45:59.650091 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:45:59.665562 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:45:59.666538 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:45:59.666853 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:45:59.667857 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:45:59.669579 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:45:59.669723 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.669862 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:45:59.669950 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:45:59.705020 23612 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.131:36475
I20260619 18:45:59.705145 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.131:36475 every 8 connection(s)
I20260619 18:45:59.706653 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/data/info.pb
I20260619 18:45:59.710933 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23612
I20260619 18:45:59.711138 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal/instance
I20260619 18:45:59.711738 3414 external_mini_cluster.cc:1428] Setting key 72c89bde0612c1d33f4d9189f444c72a
I20260619 18:45:59.721158 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.709560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.131:51523 (local address 127.3.85.190:36987)
0619 18:45:59.709800 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.709808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.711676 (+ 1868us) server_negotiation.cc:407] Connection header received
0619 18:45:59.714128 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.714132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.714204 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.714285 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:59.715696 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.716452 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.718040 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.718514 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.718703 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.720244 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:59.720257 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:59.720260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:59.720282 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:59.720314 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:59.720326 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:59.720424 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.720668 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.720774 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:45:59.722198 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.710239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:36987 (local address 127.3.85.131:51523)
0619 18:45:59.711484 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:59.711530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:59.713781 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:59.714614 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:59.714627 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:59.714761 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:45:59.715489 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.715509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.716604 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.716608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.717876 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.717888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.718801 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.718809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.719061 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.719876 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:45:59.719879 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:45:59.719923 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:45:59.720008 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:45:59.720424 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:45:59.720438 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:45:59.720448 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:45:59.720652 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 18:45:59.721104 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":425,"threads_started":1}
I20260619 18:45:59.723668 23739 heartbeater.cc:347] Connected to a master server at 127.3.85.190:36987
I20260619 18:45:59.724014 23739 heartbeater.cc:464] Registering TS with master...
I20260619 18:45:59.724840 23739 heartbeater.cc:511] Master 127.3.85.190:36987 requested a full tablet report, sending...
I20260619 18:45:59.726598 23291 ts_manager.cc:194] Registered new tserver with Master: c96a388eb0634ae58df903191d9f4ecf (127.3.85.131:36475)
I20260619 18:45:59.727803 3414 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:45:59.727830 23291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.85.131:51523
I20260619 18:45:59.738009 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.732032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.3.85.190:36987)
0619 18:45:59.732216 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.732224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.732568 (+ 344us) server_negotiation.cc:407] Connection header received
0619 18:45:59.732712 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.732716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.732779 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.732880 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:45:59.733325 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.734007 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.735515 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.735970 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.736129 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.737255 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:45:59.737261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:45:59.737264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:45:59.737284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:45:59.737317 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:45:59.737336 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:45:59.737442 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.737589 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.737699 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:45:59.742792 23291 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:45:59.749782 23291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:45:59.765898 23478 heartbeater.cc:503] Master 127.3.85.190:36987 was elected leader, sending a full tablet report...
I20260619 18:45:59.776757 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.763666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.3.85.130:45325)
0619 18:45:59.764815 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.764829 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.767459 (+ 2630us) server_negotiation.cc:407] Connection header received
0619 18:45:59.767799 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.767807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.768159 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.768351 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:59.769705 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.771375 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.773772 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.775306 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.775481 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.775748 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.776132 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.776344 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":874,"threads_started":1}
I20260619 18:45:59.777292 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.763156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.3.85.129:39695)
0619 18:45:59.767897 (+ 4741us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.767908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.767941 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:45:59.768043 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.768050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.768300 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.768526 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:59.770239 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.771479 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.774358 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.776092 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.776314 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.776598 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.776718 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.776906 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4547,"thread_start_us":4425,"threads_started":1}
I20260619 18:45:59.777336 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.763071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:49090)
0619 18:45:59.765551 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:59.765610 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:59.767261 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:59.768774 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:59.768780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:59.768811 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:59.769464 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.769482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.771625 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.771634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.774115 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.774131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.776353 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.776359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.776534 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.776570 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:45:59.776672 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:45:59.776901 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"mutex_wait_us":50,"thread_start_us":1954,"threads_started":1}
I20260619 18:45:59.777738 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.768121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:36475 (local address 127.0.0.1:41066)
0619 18:45:59.768884 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:59.768903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:59.769110 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:59.770036 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:59.770044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:59.770073 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:59.770519 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.770532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.772476 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.772486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.774710 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.774724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.776914 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.776920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.777097 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.777128 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:45:59.777244 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:45:59.777420 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":481,"threads_started":1}
I20260619 18:45:59.778065 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.763165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45325 (local address 127.0.0.1:59432)
0619 18:45:59.767295 (+ 4130us) negotiation.cc:107] Waiting for socket to connect
0619 18:45:59.767319 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:45:59.767601 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:45:59.768699 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:45:59.768711 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:45:59.768750 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:45:59.769496 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.769510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.771568 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.771573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.773563 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:45:59.773575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.775543 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.775550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:45:59.775747 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.775825 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 18:45:59.777448 (+ 1623us) client_negotiation.cc:240] Negotiation successful
0619 18:45:59.777611 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3953,"mutex_wait_us":149,"thread_start_us":3760,"threads_started":1}
I20260619 18:45:59.778656 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:45:59.767704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.3.85.131:36475)
0619 18:45:59.768455 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 18:45:59.768462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:45:59.768996 (+ 534us) server_negotiation.cc:407] Connection header received
0619 18:45:59.769322 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:45:59.769332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:45:59.769568 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:45:59.769725 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:45:59.770735 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.772318 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.774951 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:45:59.776393 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:45:59.776521 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:45:59.776781 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:45:59.778050 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0619 18:45:59.778244 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":539,"threads_started":1}
I20260619 18:45:59.781669 23544 tablet_service.cc:1467] Processing CreateTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 (DEFAULT_TABLE table=test-table [id=0d0ef02523b04331af5b613314dfc646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:59.781868 23413 tablet_service.cc:1467] Processing CreateTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 (DEFAULT_TABLE table=test-table [id=0d0ef02523b04331af5b613314dfc646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:59.781870 23674 tablet_service.cc:1467] Processing CreateTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 (DEFAULT_TABLE table=test-table [id=0d0ef02523b04331af5b613314dfc646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:45:59.783155 23544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b09d8a6bc0497b8d17015a98e53c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:59.783277 23413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b09d8a6bc0497b8d17015a98e53c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:59.783288 23674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b09d8a6bc0497b8d17015a98e53c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:45:59.795895 23757 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Bootstrap starting.
I20260619 18:45:59.798483 23758 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Bootstrap starting.
I20260619 18:45:59.799362 23759 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Bootstrap starting.
I20260619 18:45:59.799948 23757 tablet_bootstrap.cc:654] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:59.801375 23757 log.cc:824] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:59.802839 23758 tablet_bootstrap.cc:654] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:59.803423 23759 tablet_bootstrap.cc:654] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Neither blocks nor log segments found. Creating new log.
I20260619 18:45:59.804081 23758 log.cc:824] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:59.804991 23759 log.cc:824] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:45:59.807458 23758 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: No bootstrap required, opened a new log
I20260619 18:45:59.807700 23759 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: No bootstrap required, opened a new log
I20260619 18:45:59.807796 23758 ts_tablet_manager.cc:1397] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:45:59.808009 23759 ts_tablet_manager.cc:1397] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:45:59.808279 23757 tablet_bootstrap.cc:492] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: No bootstrap required, opened a new log
I20260619 18:45:59.808564 23757 ts_tablet_manager.cc:1397] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260619 18:45:59.816032 23759 raft_consensus.cc:348] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.816493 23759 raft_consensus.cc:374] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:59.816637 23759 raft_consensus.cc:729] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6f3a27d7e74641b32a09326c61bbd2, State: Initialized, Role: FOLLOWER
I20260619 18:45:59.817332 23759 consensus_queue.cc:260] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.819298 23759 ts_tablet_manager.cc:1428] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 18:45:59.819765 23757 raft_consensus.cc:348] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.819980 23758 raft_consensus.cc:348] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.820273 23757 raft_consensus.cc:374] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:59.820381 23757 raft_consensus.cc:729] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0202aa3a337f45edb478e1c4033676c0, State: Initialized, Role: FOLLOWER
I20260619 18:45:59.820415 23758 raft_consensus.cc:374] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:45:59.820503 23758 raft_consensus.cc:729] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96a388eb0634ae58df903191d9f4ecf, State: Initialized, Role: FOLLOWER
I20260619 18:45:59.821091 23758 consensus_queue.cc:260] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.821092 23757 consensus_queue.cc:260] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:45:59.823076 23609 heartbeater.cc:503] Master 127.3.85.190:36987 was elected leader, sending a full tablet report...
I20260619 18:45:59.823531 23758 ts_tablet_manager.cc:1428] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260619 18:45:59.823699 23757 ts_tablet_manager.cc:1428] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 18:45:59.824165 23739 heartbeater.cc:503] Master 127.3.85.190:36987 was elected leader, sending a full tablet report...
W20260619 18:45:59.958513 23740 tablet.cc:2369] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:59.982728 23610 tablet.cc:2369] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:45:59.989323 23479 tablet.cc:2369] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:00.023995 23763 raft_consensus.cc:482] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:00.024291 23763 raft_consensus.cc:504] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:46:00.025961 23763 leader_election.cc:288] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0202aa3a337f45edb478e1c4033676c0 (127.3.85.130:45325), c96a388eb0634ae58df903191d9f4ecf (127.3.85.131:36475)
I20260619 18:46:00.032720 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.026426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:42933 (local address 127.3.85.130:45325)
0619 18:46:00.026634 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.026639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.027063 (+ 424us) server_negotiation.cc:407] Connection header received
0619 18:46:00.027293 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.027299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.027414 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.027527 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:00.028233 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.028943 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.030965 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.031735 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.031953 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.032030 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.032300 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.032453 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:46:00.032879 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.026276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.130:45325 (local address 127.3.85.129:42933)
0619 18:46:00.026948 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.026968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:00.027116 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:00.027639 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:00.027645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:00.027680 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:00.028067 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:00.028076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.029085 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.029090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:00.030766 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:00.030779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.031954 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.031960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:00.032132 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.032157 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:46:00.032261 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:46:00.032430 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"spinlock_wait_cycles":29952,"thread_start_us":325,"threads_started":1}
I20260619 18:46:00.033310 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.027343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.85.129:38081 (local address 127.3.85.131:36475)
0619 18:46:00.027543 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.027548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.027799 (+ 251us) server_negotiation.cc:407] Connection header received
0619 18:46:00.028034 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.028039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.028100 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.028198 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:00.028696 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.029528 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.031476 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.032322 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.032504 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.032575 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.032911 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.033049 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:46:00.033973 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.027061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.131:36475 (local address 127.3.85.129:38081)
0619 18:46:00.027690 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.027704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:00.027885 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:00.028220 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:00.028224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:00.028246 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:00.028547 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:00.028555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.029666 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.029671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:00.031249 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:00.031262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.032532 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.032539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:00.032708 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.032730 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:46:00.033504 (+ 774us) client_negotiation.cc:240] Negotiation successful
0619 18:46:00.033661 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":332,"threads_started":1}
I20260619 18:46:00.034080 23564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b09d8a6bc0497b8d17015a98e53c72" candidate_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0202aa3a337f45edb478e1c4033676c0" is_pre_election: true
I20260619 18:46:00.034741 23564 raft_consensus.cc:2454] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6f3a27d7e74641b32a09326c61bbd2 in term 0.
I20260619 18:46:00.035226 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b09d8a6bc0497b8d17015a98e53c72" candidate_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96a388eb0634ae58df903191d9f4ecf" is_pre_election: true
I20260619 18:46:00.035518 23369 leader_election.cc:302] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0202aa3a337f45edb478e1c4033676c0, 6f6f3a27d7e74641b32a09326c61bbd2; no voters:
I20260619 18:46:00.035722 23694 raft_consensus.cc:2454] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f6f3a27d7e74641b32a09326c61bbd2 in term 0.
I20260619 18:46:00.035971 23763 raft_consensus.cc:2781] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:00.036115 23763 raft_consensus.cc:482] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:00.036224 23763 raft_consensus.cc:3037] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:00.038514 23763 raft_consensus.cc:504] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:46:00.039202 23763 leader_election.cc:288] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [CANDIDATE]: Term 1 election: Requested vote from peers 0202aa3a337f45edb478e1c4033676c0 (127.3.85.130:45325), c96a388eb0634ae58df903191d9f4ecf (127.3.85.131:36475)
I20260619 18:46:00.039940 23694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b09d8a6bc0497b8d17015a98e53c72" candidate_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96a388eb0634ae58df903191d9f4ecf"
I20260619 18:46:00.039940 23564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87b09d8a6bc0497b8d17015a98e53c72" candidate_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0202aa3a337f45edb478e1c4033676c0"
I20260619 18:46:00.040181 23564 raft_consensus.cc:3037] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:00.040181 23694 raft_consensus.cc:3037] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:00.043605 23564 raft_consensus.cc:2454] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6f3a27d7e74641b32a09326c61bbd2 in term 1.
I20260619 18:46:00.043605 23694 raft_consensus.cc:2454] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f6f3a27d7e74641b32a09326c61bbd2 in term 1.
I20260619 18:46:00.044123 23369 leader_election.cc:302] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0202aa3a337f45edb478e1c4033676c0, 6f6f3a27d7e74641b32a09326c61bbd2; no voters:
I20260619 18:46:00.044435 23763 raft_consensus.cc:2781] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:00.045099 23763 raft_consensus.cc:686] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f6f3a27d7e74641b32a09326c61bbd2, State: Running, Role: LEADER
I20260619 18:46:00.045838 23763 consensus_queue.cc:237] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } }
I20260619 18:46:00.051406 23292 catalog_manager.cc:5495] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f6f3a27d7e74641b32a09326c61bbd2 (127.3.85.129). New cstate: current_term: 1 leader_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f6f3a27d7e74641b32a09326c61bbd2" member_type: VOTER last_known_addr { host: "127.3.85.129" port: 39695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 } health_report { overall_health: UNKNOWN } } }
I20260619 18:46:00.071166 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.065472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.3.85.129:39695)
0619 18:46:00.065660 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.065664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.065850 (+ 186us) server_negotiation.cc:407] Connection header received
0619 18:46:00.066049 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.066054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.066153 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.066295 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:00.066763 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.067478 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.068876 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.069324 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.069438 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.070019 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:00.070657 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:00.070770 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.070893 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.070965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:46:00.080803 23694 raft_consensus.cc:1261] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 1 FOLLOWER]: Refusing update from remote peer 6f6f3a27d7e74641b32a09326c61bbd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:00.080803 23564 raft_consensus.cc:1261] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 1 FOLLOWER]: Refusing update from remote peer 6f6f3a27d7e74641b32a09326c61bbd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:00.081990 23768 consensus_queue.cc:1035] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0202aa3a337f45edb478e1c4033676c0" member_type: VOTER last_known_addr { host: "127.3.85.130" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:00.082381 23763 consensus_queue.cc:1035] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96a388eb0634ae58df903191d9f4ecf" member_type: VOTER last_known_addr { host: "127.3.85.131" port: 36475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:00.095788 23771 mvcc.cc:205] Tried to move back new op lower bound from 7298640937279049728 to 7298640937152831488. Current Snapshot: MvccSnapshot[applied={T|T < 7298640937279049728}]
I20260619 18:46:00.115015 23291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:00.115307 23291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:46:00.119156 23291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f3e85861e774697a1e044539871eb90: Sending DeleteTablet for 3 replicas of tablet 87b09d8a6bc0497b8d17015a98e53c72
I20260619 18:46:00.120553 23674 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:00 UTC) from {username='slave'} at 127.0.0.1:41066
I20260619 18:46:00.120555 23413 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:00 UTC) from {username='slave'} at 127.0.0.1:49090
I20260619 18:46:00.121012 23544 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b09d8a6bc0497b8d17015a98e53c72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:46:00 UTC) from {username='slave'} at 127.0.0.1:59432
I20260619 18:46:00.122831 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23351
I20260619 18:46:00.163142 23782 tablet_replica.cc:330] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: stopping tablet replica
I20260619 18:46:00.164032 23782 raft_consensus.cc:2229] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:00.165817 23784 tablet_replica.cc:330] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: stopping tablet replica
I20260619 18:46:00.166662 23782 raft_consensus.cc:2258] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:00.169543 23784 raft_consensus.cc:2229] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:46:00.170132 23784 raft_consensus.cc:2258] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:00.171999 23783 tablet_replica.cc:330] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: stopping tablet replica
I20260619 18:46:00.172503 23783 raft_consensus.cc:2229] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:46:00.173290 23783 raft_consensus.cc:2258] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:46:00.173445 23784 ts_tablet_manager.cc:1905] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:00.173545 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.124784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.3.85.129:39695)
0619 18:46:00.125048 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.125057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.156892 (+ 31835us) server_negotiation.cc:407] Connection header received
0619 18:46:00.157537 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.157546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.157720 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.157829 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:00.158857 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.160493 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.167023 (+ 6530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.168073 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.168233 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.171377 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:00.171395 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:00.171417 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:00.172343 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:00.172426 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:00.172459 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:00.172594 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.172926 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.173063 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:46:00.175634 23473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:00.177214 23782 ts_tablet_manager.cc:1905] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:00.182014 23784 ts_tablet_manager.cc:1918] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:00.182232 23784 log.cc:1196] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-1/wal/wals/87b09d8a6bc0497b8d17015a98e53c72
I20260619 18:46:00.182850 23784 ts_tablet_manager.cc:1939] T 87b09d8a6bc0497b8d17015a98e53c72 P 0202aa3a337f45edb478e1c4033676c0: Deleting consensus metadata
I20260619 18:46:00.184692 23279 catalog_manager.cc:4841] TS 0202aa3a337f45edb478e1c4033676c0 (127.3.85.130:45325): tablet 87b09d8a6bc0497b8d17015a98e53c72 (table test-table [id=0d0ef02523b04331af5b613314dfc646]) successfully deleted
I20260619 18:46:00.185915 23782 ts_tablet_manager.cc:1918] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:46:00.186110 23782 log.cc:1196] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Deleting WAL directory at /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894429649633-3414-0/minicluster-data/ts-2/wal/wals/87b09d8a6bc0497b8d17015a98e53c72
I20260619 18:46:00.186667 23782 ts_tablet_manager.cc:1939] T 87b09d8a6bc0497b8d17015a98e53c72 P c96a388eb0634ae58df903191d9f4ecf: Deleting consensus metadata
I20260619 18:46:00.188768 23279 catalog_manager.cc:4841] TS c96a388eb0634ae58df903191d9f4ecf (127.3.85.131:36475): tablet 87b09d8a6bc0497b8d17015a98e53c72 (table test-table [id=0d0ef02523b04331af5b613314dfc646]) successfully deleted
I20260619 18:46:00.176944 23783 ts_tablet_manager.cc:1905] T 87b09d8a6bc0497b8d17015a98e53c72 P 6f6f3a27d7e74641b32a09326c61bbd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:46:00.281745 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23351
I20260619 18:46:00.301079 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23482
W20260619 18:46:00.301270 23276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.85.129:39695 (error 108)
I20260619 18:46:00.303377 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.302652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:49114)
0619 18:46:00.303092 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.303207 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
W20260619 18:46:00.303653 23276 catalog_manager.cc:4584] TS 6f6f3a27d7e74641b32a09326c61bbd2 (127.3.85.129:39695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87b09d8a6bc0497b8d17015a98e53c72: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
I20260619 18:46:00.308593 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.301969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.3.85.130:45325)
0619 18:46:00.302158 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.302165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.302371 (+ 206us) server_negotiation.cc:407] Connection header received
0619 18:46:00.302507 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.302514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.302627 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.302717 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:00.303252 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.304176 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.305491 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.305915 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.306061 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.307132 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:00.307142 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:00.307157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:00.307780 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:00.307831 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:00.307853 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:00.307986 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.308131 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.308251 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:46:00.309347 23604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:00.357161 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.356685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36912)
0619 18:46:00.356924 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.357011 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:46:00.358295 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.357880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36928)
0619 18:46:00.358068 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.358141 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:46:00.400615 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23482
I20260619 18:46:00.417779 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23612
I20260619 18:46:00.419180 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.418607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36940)
0619 18:46:00.418871 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.418983 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260619 18:46:00.420480 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.419992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36948)
0619 18:46:00.420234 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.420323 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 18:46:00.424926 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.418699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.3.85.131:36475)
0619 18:46:00.418901 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:00.418907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:00.418931 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:46:00.419035 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:00.419040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:00.419155 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:00.419294 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:00.419830 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.420726 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.422027 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:00.422622 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:00.422731 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:00.423694 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:00.423702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:00.423717 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:00.424292 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:00.424341 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:00.424360 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:00.424456 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:00.424569 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 18:46:00.424673 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:46:00.425602 23734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:00.501461 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.500926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36958)
0619 18:46:00.501202 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.501301 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 18:46:00.502791 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:00.502256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.129:39695 (local address 127.0.0.1:36970)
0619 18:46:00.502552 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:00.502628 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.129:39695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 18:46:00.517228 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23612
I20260619 18:46:00.535763 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23261
I20260619 18:46:00.536911 23322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:00.648217 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23261
2026-06-19T18:46:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3055 ms)
[----------] 6 tests from EncryptionPolicyTest (23087 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:00 dist-test-slave-09fp krb5kdc[23794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:00 dist-test-slave-09fp krb5kdc[23794](info): set up 2 sockets
Jun 19 18:46:00 dist-test-slave-09fp krb5kdc[23794](info): commencing operation
krb5kdc: starting...
W20260619 18:46:01.783016 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:01 dist-test-slave-09fp krb5kdc[23794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:02.063190 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:02.432376 23807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:02.432693 23807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:02.432862 23807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:02.440454 23807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:02.440582 23807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:02.440682 23807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:02.440800 23807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:02.440866 23807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:02.451843 23807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:02.453264 23807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:02.454743 23807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:02.467068 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:02.467147 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:02.467177 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:02.468240 23807 server_base.cc:1034] running on GCE node
I20260619 18:46:02.469074 23807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:02.470629 23807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:02.471848 23807 hybrid_clock.cc:648] HybridClock initialized: now 1781894762471792 us; error 49 us; skew 500 ppm
I20260619 18:46:02.472189 23807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:02 dist-test-slave-09fp krb5kdc[23794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:02.475752 23807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:02.477682 23807 webserver.cc:460] Webserver started at http://127.0.0.1:44871/ using document root <none> and password file <none>
I20260619 18:46:02.478611 23807 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:02.478772 23807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:02.479108 23807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:02.482431 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "cfea97ab22d141a387ade2b8f7739580"
format_stamp: "Formatted at 2026-06-19 18:46:02 on dist-test-slave-09fp"
server_key: "a9cd2ac638cb3d1645bffca8f4e5d268"
server_key_iv: "81a63d3f64d18f0edb7d28463b97f5d0"
server_key_version: "encryptionkey@0"
I20260619 18:46:02.483433 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "cfea97ab22d141a387ade2b8f7739580"
format_stamp: "Formatted at 2026-06-19 18:46:02 on dist-test-slave-09fp"
server_key: "a9cd2ac638cb3d1645bffca8f4e5d268"
server_key_iv: "81a63d3f64d18f0edb7d28463b97f5d0"
server_key_version: "encryptionkey@0"
I20260619 18:46:02.489300 23807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:02.493443 23822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:02.494975 23807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:02.495240 23807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "cfea97ab22d141a387ade2b8f7739580"
format_stamp: "Formatted at 2026-06-19 18:46:02 on dist-test-slave-09fp"
server_key: "a9cd2ac638cb3d1645bffca8f4e5d268"
server_key_iv: "81a63d3f64d18f0edb7d28463b97f5d0"
server_key_version: "encryptionkey@0"
I20260619 18:46:02.495496 23807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:02.515271 23807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:02.519909 23807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:02.520238 23807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:02.540683 23807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45561
I20260619 18:46:02.540957 23873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45561 every 8 connection(s)
I20260619 18:46:02.542204 23807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:02.546132 23874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:02.546738 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23807
I20260619 18:46:02.546952 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:02.547595 3414 external_mini_cluster.cc:1428] Setting key 83e700ec12e1173c6f95d682decff842
I20260619 18:46:02.561352 23874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580: Bootstrap starting.
Jun 19 18:46:02 dist-test-slave-09fp krb5kdc[23794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:02.565142 23874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:02.566289 23874 log.cc:824] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:02.570182 23874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580: No bootstrap required, opened a new log
I20260619 18:46:02.575176 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:02.549791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46979 (local address 127.0.0.1:45561)
0619 18:46:02.550693 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:02.550709 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:02.550760 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:46:02.553267 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:02.553300 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:02.553899 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:02.554400 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:02.556009 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:02.558095 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:02.559622 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:02.560685 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:02.560885 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:02.565534 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:02.565543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:02.565563 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:02.565614 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:02.569652 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:02.570815 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:02.570824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:02.570834 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:02.571066 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:02.571504 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:02.571509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:02.571513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:02.572293 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:02.572659 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:02.573164 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 18:46:02.573569 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":381,"threads_started":1}
I20260619 18:46:02.579730 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:02.580639 23874 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 18:46:02.581115 23874 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:02.581254 23874 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfea97ab22d141a387ade2b8f7739580, State: Initialized, Role: FOLLOWER
I20260619 18:46:02.581949 23874 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 18:46:02.582224 23874 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:02.582373 23874 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:02.582598 23874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:02.585646 23874 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 18:46:02.586275 23874 leader_election.cc:302] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfea97ab22d141a387ade2b8f7739580; no voters:
I20260619 18:46:02.587042 23874 leader_election.cc:288] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:02.587145 23884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:02.590193 23874 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:02.592144 23884 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [term 1 LEADER]: Becoming Leader. State: Replica: cfea97ab22d141a387ade2b8f7739580, State: Running, Role: LEADER
I20260619 18:46:02.593129 23884 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } }
I20260619 18:46:02.598162 23886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfea97ab22d141a387ade2b8f7739580. Latest consensus state: current_term: 1 leader_uuid: "cfea97ab22d141a387ade2b8f7739580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } } }
I20260619 18:46:02.598697 23886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:02.597816 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfea97ab22d141a387ade2b8f7739580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfea97ab22d141a387ade2b8f7739580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45561 } } }
I20260619 18:46:02.599658 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:02.604938 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:02.584984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59679 (local address 127.0.0.1:45561)
0619 18:46:02.585189 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:02.585197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:02.585657 (+ 460us) server_negotiation.cc:407] Connection header received
0619 18:46:02.585876 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:02.585882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:02.585962 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:02.586098 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:02.587905 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:02.588892 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:02.590533 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:02.591076 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:02.591164 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:02.594904 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:02.594910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:02.594913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:02.594952 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:02.599097 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:02.600638 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:02.600647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:02.600654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:02.600861 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:02.601663 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:02.601671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:02.601677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:02.602224 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:02.603860 (+ 1636us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:02.604122 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:46:02.604386 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:46:02.607999 23895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:02.613591 23895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:02.623386 23895 catalog_manager.cc:1347] Generated new cluster ID: 005c32850318464b9d19df27eacc9859
I20260619 18:46:02.623531 23895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:02.632249 23895 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:02.634261 23895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:02.643486 23895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfea97ab22d141a387ade2b8f7739580: Generated new TSK 0
I20260619 18:46:02.644567 23895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:02.657625 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23807
I20260619 18:46:02.660396 23869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:02.759104 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23807
W20260619 18:46:02.777176 23792 connection.cc:535] client connection to 127.0.0.1:45561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45561): Connection reset by peer (error 104)
2026-06-19T18:46:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2105 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:02 dist-test-slave-09fp krb5kdc[23907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:02 dist-test-slave-09fp krb5kdc[23907](info): set up 2 sockets
Jun 19 18:46:02 dist-test-slave-09fp krb5kdc[23907](info): commencing operation
krb5kdc: starting...
W20260619 18:46:03.870918 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:04 dist-test-slave-09fp krb5kdc[23907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:04.146536 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:04.495100 23920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:04.495471 23920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:04.495657 23920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:04.503477 23920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:04.503618 23920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:04.503715 23920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:04.503830 23920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:04.503922 23920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:04.514364 23920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:04.515841 23920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:04.517212 23920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:04.529776 23925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:04.529808 23926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:04.529637 23928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:04.531214 23920 server_base.cc:1034] running on GCE node
I20260619 18:46:04.532166 23920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:04.533735 23920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:04.535027 23920 hybrid_clock.cc:648] HybridClock initialized: now 1781894764534974 us; error 46 us; skew 500 ppm
I20260619 18:46:04.535459 23920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:04 dist-test-slave-09fp krb5kdc[23907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:04.542706 23920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:04.544513 23920 webserver.cc:460] Webserver started at http://127.0.0.1:45489/ using document root <none> and password file <none>
I20260619 18:46:04.545265 23920 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:04.545380 23920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:04.545655 23920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:04.548952 23920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "6334ff64324a4f09aede9e871824b1d2"
format_stamp: "Formatted at 2026-06-19 18:46:04 on dist-test-slave-09fp"
server_key: "1727e377d7a7fee8446b9f92a7e0759a"
server_key_iv: "3bd0eea2d01537b3f362f55dd9eeb9d6"
server_key_version: "encryptionkey@0"
I20260619 18:46:04.549777 23920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "6334ff64324a4f09aede9e871824b1d2"
format_stamp: "Formatted at 2026-06-19 18:46:04 on dist-test-slave-09fp"
server_key: "1727e377d7a7fee8446b9f92a7e0759a"
server_key_iv: "3bd0eea2d01537b3f362f55dd9eeb9d6"
server_key_version: "encryptionkey@0"
I20260619 18:46:04.555650 23920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 18:46:04.559643 23935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:04.561173 23920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:04.561349 23920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "6334ff64324a4f09aede9e871824b1d2"
format_stamp: "Formatted at 2026-06-19 18:46:04 on dist-test-slave-09fp"
server_key: "1727e377d7a7fee8446b9f92a7e0759a"
server_key_iv: "3bd0eea2d01537b3f362f55dd9eeb9d6"
server_key_version: "encryptionkey@0"
I20260619 18:46:04.561553 23920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:04.591640 23920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:04.596258 23920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:04.596554 23920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:04.617007 23920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37573
I20260619 18:46:04.617033 23986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37573 every 8 connection(s)
I20260619 18:46:04.618695 23920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:04.622975 23987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:04.623750 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 23920
I20260619 18:46:04.623965 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:04.624503 3414 external_mini_cluster.cc:1428] Setting key 3d0dc95dfd8dd4c26e41b5b88dca5fb0
I20260619 18:46:04.638195 23987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2: Bootstrap starting.
I20260619 18:46:04.639577 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:04.626551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49093 (local address 127.0.0.1:37573)
0619 18:46:04.627446 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:04.627458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:04.627506 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:46:04.629691 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:04.629722 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:04.630319 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:04.630772 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:04.631399 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:04.632759 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:04.634793 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:04.635615 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:04.635762 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:04.637154 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:04.637162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:04.637178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:04.637753 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:04.637836 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:04.637863 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:04.637991 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:04.638327 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:46:04.638545 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":434,"threads_started":1}
I20260619 18:46:04.642026 23987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:04.643363 23987 log.cc:824] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:04.643911 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:04.646389 23987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2: No bootstrap required, opened a new log
I20260619 18:46:04.655226 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:04.648562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33477 (local address 127.0.0.1:37573)
0619 18:46:04.648766 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:04.648775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:04.649181 (+ 406us) server_negotiation.cc:407] Connection header received
0619 18:46:04.649423 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:04.649427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:04.649495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:04.649626 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:04.650129 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:04.650990 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:04.652428 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:04.652920 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:04.653098 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:04.654365 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:04.654370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:04.654374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:04.654393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:04.654497 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:04.654519 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:04.654652 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:04.654848 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:46:04.654907 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:46:04.655146 23987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } }
I20260619 18:46:04.655787 23987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:04.655929 23987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6334ff64324a4f09aede9e871824b1d2, State: Initialized, Role: FOLLOWER
I20260619 18:46:04.656708 23987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } }
I20260619 18:46:04.657058 23987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:04.657218 23987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:04.657356 23987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:04.659766 23987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } }
I20260619 18:46:04.660396 23987 leader_election.cc:302] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6334ff64324a4f09aede9e871824b1d2; no voters:
I20260619 18:46:04.661015 23987 leader_election.cc:288] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:04.661208 23997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:04.661908 23997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [term 1 LEADER]: Becoming Leader. State: Replica: 6334ff64324a4f09aede9e871824b1d2, State: Running, Role: LEADER
I20260619 18:46:04.662707 23997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } }
I20260619 18:46:04.663928 23987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:04.665871 23999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6334ff64324a4f09aede9e871824b1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } } }
I20260619 18:46:04.666308 23999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:04.666654 23998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6334ff64324a4f09aede9e871824b1d2. Latest consensus state: current_term: 1 leader_uuid: "6334ff64324a4f09aede9e871824b1d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6334ff64324a4f09aede9e871824b1d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37573 } } }
I20260619 18:46:04.667125 23998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:04.669190 24004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:04.676028 24004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:04.685544 24004 catalog_manager.cc:1347] Generated new cluster ID: eecd3f019be843b59eb6c42cbdf7f2e5
I20260619 18:46:04.685695 24004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:04.711989 24004 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:04.713712 24004 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:04.724287 24004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6334ff64324a4f09aede9e871824b1d2: Generated new TSK 0
I20260619 18:46:04.725162 24004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:04.734431 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23920
W20260619 18:46:04.735652 23982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49093
W20260619 18:46:04.736588 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 23920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:04.736794 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 23920
W20260619 18:46:04.754595 23905 connection.cc:535] client connection to 127.0.0.1:37573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37573): Connection reset by peer (error 104)
2026-06-19T18:46:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:04 dist-test-slave-09fp krb5kdc[24018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:04 dist-test-slave-09fp krb5kdc[24018](info): set up 2 sockets
Jun 19 18:46:04 dist-test-slave-09fp krb5kdc[24018](info): commencing operation
krb5kdc: starting...
W20260619 18:46:05.858964 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:06 dist-test-slave-09fp krb5kdc[24018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:06.130721 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:06.478325 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:06.478750 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:06.478937 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:06.486394 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:06.486626 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:06.486742 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:06.486858 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:06.486943 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:06.498067 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:06.499572 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:06.501044 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:06.514292 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:06.514376 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:06.514367 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:06.515157 24031 server_base.cc:1034] running on GCE node
I20260619 18:46:06.516285 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:06.517959 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:06.519233 24031 hybrid_clock.cc:648] HybridClock initialized: now 1781894766519180 us; error 56 us; skew 500 ppm
I20260619 18:46:06.519636 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:06 dist-test-slave-09fp krb5kdc[24018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:06.523231 24031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:06.524917 24031 webserver.cc:460] Webserver started at http://127.0.0.1:43353/ using document root <none> and password file <none>
I20260619 18:46:06.525703 24031 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:06.525867 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:06.526224 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:06.529539 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "16a28b7314804b2bb92121d0be58fb00"
format_stamp: "Formatted at 2026-06-19 18:46:06 on dist-test-slave-09fp"
server_key: "8ef1499bc4e844a66e2a83ee955d970c"
server_key_iv: "d4bacfde4550b0eea857c714e3133dae"
server_key_version: "encryptionkey@0"
I20260619 18:46:06.530404 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "16a28b7314804b2bb92121d0be58fb00"
format_stamp: "Formatted at 2026-06-19 18:46:06 on dist-test-slave-09fp"
server_key: "8ef1499bc4e844a66e2a83ee955d970c"
server_key_iv: "d4bacfde4550b0eea857c714e3133dae"
server_key_version: "encryptionkey@0"
I20260619 18:46:06.536285 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:06.540648 24046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:06.542640 24031 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 18:46:06.542889 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "16a28b7314804b2bb92121d0be58fb00"
format_stamp: "Formatted at 2026-06-19 18:46:06 on dist-test-slave-09fp"
server_key: "8ef1499bc4e844a66e2a83ee955d970c"
server_key_iv: "d4bacfde4550b0eea857c714e3133dae"
server_key_version: "encryptionkey@0"
I20260619 18:46:06.543140 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:06.549713 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:06.554713 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:06.555075 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:06.575335 24031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45173
I20260619 18:46:06.575392 24097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45173 every 8 connection(s)
I20260619 18:46:06.576885 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:06.577843 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24031
I20260619 18:46:06.578045 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:06.578667 3414 external_mini_cluster.cc:1428] Setting key a4db63b1eec26e8c4400a9c4bf77bd26
I20260619 18:46:06.581987 24098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:06.589529 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.580948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37635 (local address 127.0.0.1:45173)
0619 18:46:06.582081 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:06.582101 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:06.582545 (+ 444us) server_negotiation.cc:407] Connection header received
0619 18:46:06.585106 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:06.585138 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:06.585682 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:06.586196 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:06.587298 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:06.587307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:06.587327 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:06.587356 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:06.587452 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:06.587485 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:06.587738 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:06.587878 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:46:06.588114 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":485,"threads_started":1}
I20260619 18:46:06.594349 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:06.597769 24098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00: Bootstrap starting.
I20260619 18:46:06.602126 24098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:06.602689 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:06.599891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37519 (local address 127.0.0.1:45173)
0619 18:46:06.600114 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:06.600123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:06.600540 (+ 417us) server_negotiation.cc:407] Connection header received
0619 18:46:06.600694 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:06.600702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:06.600771 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:06.600896 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:06.601857 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:06.601866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:06.601873 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:06.601896 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:06.601949 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:06.601977 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:06.602122 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:06.602260 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 18:46:06.602327 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:46:06.603719 24098 log.cc:824] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:06.606842 24098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00: No bootstrap required, opened a new log
I20260619 18:46:06.614652 24098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } }
I20260619 18:46:06.615085 24098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:06.615155 24098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a28b7314804b2bb92121d0be58fb00, State: Initialized, Role: FOLLOWER
I20260619 18:46:06.615873 24098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } }
I20260619 18:46:06.616200 24098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:06.616364 24098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:06.616580 24098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:06.618912 24098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } }
I20260619 18:46:06.619442 24098 leader_election.cc:302] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a28b7314804b2bb92121d0be58fb00; no voters:
I20260619 18:46:06.620075 24098 leader_election.cc:288] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:06.620281 24108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:06.621013 24108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [term 1 LEADER]: Becoming Leader. State: Replica: 16a28b7314804b2bb92121d0be58fb00, State: Running, Role: LEADER
I20260619 18:46:06.621627 24108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } }
I20260619 18:46:06.622433 24098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:06.624543 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a28b7314804b2bb92121d0be58fb00. Latest consensus state: current_term: 1 leader_uuid: "16a28b7314804b2bb92121d0be58fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } } }
I20260619 18:46:06.625033 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:06.625073 24109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a28b7314804b2bb92121d0be58fb00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a28b7314804b2bb92121d0be58fb00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45173 } } }
I20260619 18:46:06.625480 24109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:06.627630 24114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:06.633541 24114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:06.643822 24114 catalog_manager.cc:1347] Generated new cluster ID: 7b42da45d8434ef2b42976a8e4947c12
I20260619 18:46:06.643956 24114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:06.675127 24114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:06.675622 24114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:06.686045 24114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a28b7314804b2bb92121d0be58fb00: Generated new TSK 0
I20260619 18:46:06.687031 24114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:06.693367 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24031
W20260619 18:46:06.694629 24093 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37635
W20260619 18:46:06.695420 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24031: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:06.695580 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24031
2026-06-19T18:46:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:06 dist-test-slave-09fp krb5kdc[24129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:06 dist-test-slave-09fp krb5kdc[24129](info): set up 2 sockets
Jun 19 18:46:06 dist-test-slave-09fp krb5kdc[24129](info): commencing operation
krb5kdc: starting...
W20260619 18:46:07.811141 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:07 dist-test-slave-09fp krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:08.087221 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:08.444734 24142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:08.445134 24142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:08.445294 24142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:08.452941 24142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:08.453126 24142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:08.453246 24142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:08.453358 24142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:08.453442 24142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:08.463881 24142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:08.465337 24142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:08.466759 24142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:08.479341 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:08.479417 24150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:08.479417 24147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:08.480840 24142 server_base.cc:1034] running on GCE node
I20260619 18:46:08.481674 24142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:08.483276 24142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:08.484494 24142 hybrid_clock.cc:648] HybridClock initialized: now 1781894768484453 us; error 72 us; skew 500 ppm
I20260619 18:46:08.484876 24142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:08 dist-test-slave-09fp krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:08.488266 24142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:08.489992 24142 webserver.cc:460] Webserver started at http://127.0.0.1:41847/ using document root <none> and password file <none>
I20260619 18:46:08.490871 24142 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:08.491031 24142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:08.491365 24142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:08.494709 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "3ba40daeef964661828551a4e6a61033"
format_stamp: "Formatted at 2026-06-19 18:46:08 on dist-test-slave-09fp"
server_key: "9fb7569b4919b98f8333888beebf9908"
server_key_iv: "74d4914e756e52f39eaebdc88d45cbd0"
server_key_version: "encryptionkey@0"
I20260619 18:46:08.495595 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "3ba40daeef964661828551a4e6a61033"
format_stamp: "Formatted at 2026-06-19 18:46:08 on dist-test-slave-09fp"
server_key: "9fb7569b4919b98f8333888beebf9908"
server_key_iv: "74d4914e756e52f39eaebdc88d45cbd0"
server_key_version: "encryptionkey@0"
I20260619 18:46:08.501453 24142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:46:08.505831 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:08.507555 24142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:46:08.507791 24142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "3ba40daeef964661828551a4e6a61033"
format_stamp: "Formatted at 2026-06-19 18:46:08 on dist-test-slave-09fp"
server_key: "9fb7569b4919b98f8333888beebf9908"
server_key_iv: "74d4914e756e52f39eaebdc88d45cbd0"
server_key_version: "encryptionkey@0"
I20260619 18:46:08.508039 24142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:08.530891 24142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:08.536015 24142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:08.536338 24142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:08.556763 24142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37523
I20260619 18:46:08.556861 24208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37523 every 8 connection(s)
I20260619 18:46:08.558286 24142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:08.562505 24209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:08.565594 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24142
I20260619 18:46:08.565788 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:08.566339 3414 external_mini_cluster.cc:1428] Setting key b59d7cb1633393a5a919a2a1c495b322
I20260619 18:46:08.576151 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033: Bootstrap starting.
I20260619 18:46:08.579766 24209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:08.580937 24209 log.cc:824] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:46:08 dist-test-slave-09fp krb5kdc[24129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:08.585503 24209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033: No bootstrap required, opened a new log
I20260619 18:46:08.594564 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:08.568461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43193 (local address 127.0.0.1:37523)
0619 18:46:08.569433 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:08.569447 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:08.569499 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:46:08.571915 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:08.571959 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:08.572687 (+ 728us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:08.573218 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:08.574979 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:08.576561 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:08.578150 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:08.579061 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:08.579329 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:08.585302 (+ 5973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:08.585312 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:08.585329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:08.585384 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:08.588973 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:08.590053 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:08.590062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:08.590069 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:08.590329 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:08.590914 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:08.590919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:08.590921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:08.591639 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:08.591906 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:08.592560 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 18:46:08.593172 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":426,"threads_started":1}
I20260619 18:46:08.595162 24209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260619 18:46:08.595597 24209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:08.595723 24209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba40daeef964661828551a4e6a61033, State: Initialized, Role: FOLLOWER
I20260619 18:46:08.596364 24209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260619 18:46:08.596655 24209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:08.596822 24209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:08.597007 24209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:08.599434 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:08.599866 24209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260619 18:46:08.600528 24209 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba40daeef964661828551a4e6a61033; no voters:
I20260619 18:46:08.601209 24209 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:08.601351 24214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:08.602010 24214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba40daeef964661828551a4e6a61033, State: Running, Role: LEADER
I20260619 18:46:08.602800 24214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } }
I20260619 18:46:08.604830 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba40daeef964661828551a4e6a61033. Latest consensus state: current_term: 1 leader_uuid: "3ba40daeef964661828551a4e6a61033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } } }
I20260619 18:46:08.605357 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:08.606007 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba40daeef964661828551a4e6a61033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba40daeef964661828551a4e6a61033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37523 } } }
I20260619 18:46:08.606441 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:08.609791 24209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:08.616166 24219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:08.623493 24219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:08.636224 24219 catalog_manager.cc:1347] Generated new cluster ID: 34f4d5278b8f49379b93cded0490c0af
I20260619 18:46:08.636476 24219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:08.640097 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:08.616778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45771 (local address 127.0.0.1:37523)
0619 18:46:08.616984 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:08.616990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:08.618369 (+ 1379us) server_negotiation.cc:407] Connection header received
0619 18:46:08.618678 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:08.618688 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:08.618831 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:08.619007 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:08.621597 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:08.622764 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:08.624755 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:08.625462 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:08.625589 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:08.629374 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:08.629382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:08.629386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:08.629437 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:08.634026 (+ 4589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:08.635487 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:08.635498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:08.635504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:08.635732 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:08.636380 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:08.636389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:08.636395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:08.637027 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:08.637375 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:08.639116 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0619 18:46:08.639514 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:46:08.654829 24219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:08.656299 24219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:08.664541 24219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba40daeef964661828551a4e6a61033: Generated new TSK 0
I20260619 18:46:08.665541 24219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:08.674003 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24142
I20260619 18:46:08.675312 24204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:08.771521 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24142
2026-06-19T18:46:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:08 dist-test-slave-09fp krb5kdc[24241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:08 dist-test-slave-09fp krb5kdc[24241](info): set up 2 sockets
Jun 19 18:46:08 dist-test-slave-09fp krb5kdc[24241](info): commencing operation
krb5kdc: starting...
W20260619 18:46:09.890981 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:10 dist-test-slave-09fp krb5kdc[24241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:10.166082 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:10.517429 24254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:10.517760 24254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:10.517935 24254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:10.525483 24254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:10.525627 24254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:10.525715 24254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:10.525815 24254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:10.525876 24254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:10.536363 24254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:10.537798 24254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:10.539245 24254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:10.552517 24259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:10.552572 24260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:10.552481 24262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:10.553289 24254 server_base.cc:1034] running on GCE node
I20260619 18:46:10.554186 24254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:10.555799 24254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:10.557057 24254 hybrid_clock.cc:648] HybridClock initialized: now 1781894770557019 us; error 63 us; skew 500 ppm
I20260619 18:46:10.557451 24254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:10 dist-test-slave-09fp krb5kdc[24241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:10.564253 24254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:10.566074 24254 webserver.cc:460] Webserver started at http://127.0.0.1:39815/ using document root <none> and password file <none>
I20260619 18:46:10.566962 24254 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:10.567123 24254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:10.567456 24254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:10.570793 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "ec2ebd58a99749f297489ff2ad54a291"
format_stamp: "Formatted at 2026-06-19 18:46:10 on dist-test-slave-09fp"
server_key: "fe0be89fc678bf1d56baf7c55de16e5b"
server_key_iv: "e2a412eecc7772c24afb111f0ce692e2"
server_key_version: "encryptionkey@0"
I20260619 18:46:10.571681 24254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "ec2ebd58a99749f297489ff2ad54a291"
format_stamp: "Formatted at 2026-06-19 18:46:10 on dist-test-slave-09fp"
server_key: "fe0be89fc678bf1d56baf7c55de16e5b"
server_key_iv: "e2a412eecc7772c24afb111f0ce692e2"
server_key_version: "encryptionkey@0"
I20260619 18:46:10.577704 24254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:46:10.581624 24269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:10.583261 24254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:46:10.583491 24254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "ec2ebd58a99749f297489ff2ad54a291"
format_stamp: "Formatted at 2026-06-19 18:46:10 on dist-test-slave-09fp"
server_key: "fe0be89fc678bf1d56baf7c55de16e5b"
server_key_iv: "e2a412eecc7772c24afb111f0ce692e2"
server_key_version: "encryptionkey@0"
I20260619 18:46:10.583729 24254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:10.611153 24254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:10.615769 24254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:10.616109 24254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:10.636237 24254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35645
I20260619 18:46:10.636269 24320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35645 every 8 connection(s)
I20260619 18:46:10.637754 24254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:10.641919 24321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:10.645364 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24254
I20260619 18:46:10.645547 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:10.645982 3414 external_mini_cluster.cc:1428] Setting key d421c2b5ec5295377c90ddef77cb4471
I20260619 18:46:10.655481 24321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291: Bootstrap starting.
I20260619 18:46:10.659535 24321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:10.660790 24321 log.cc:824] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:10.661391 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:10.648129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55471 (local address 127.0.0.1:35645)
0619 18:46:10.649017 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:10.649029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:10.649075 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:46:10.651102 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:10.651132 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:10.651641 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:10.652030 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:10.652699 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:10.654185 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:10.656162 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:10.656907 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:10.657062 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:10.658657 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:10.658666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:10.658682 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:10.659271 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:10.659366 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:10.659402 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:10.659498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:10.659809 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 18:46:10.660162 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":415,"threads_started":1}
I20260619 18:46:10.664307 24321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291: No bootstrap required, opened a new log
I20260619 18:46:10.665565 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:10.673182 24321 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } }
I20260619 18:46:10.673671 24321 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:10.673802 24321 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2ebd58a99749f297489ff2ad54a291, State: Initialized, Role: FOLLOWER
I20260619 18:46:10.674585 24321 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } }
I20260619 18:46:10.674891 24321 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:10.675068 24321 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:10.675252 24321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:10.677227 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:10.670278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48769 (local address 127.0.0.1:35645)
0619 18:46:10.670543 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:10.670552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:10.670976 (+ 424us) server_negotiation.cc:407] Connection header received
0619 18:46:10.671214 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:10.671218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:10.671295 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:10.671436 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:10.671940 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:10.672671 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:10.674189 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:10.674730 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:10.674888 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:10.676357 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:10.676363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:10.676366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:10.676394 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:10.676446 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:10.676472 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:10.676599 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:10.676782 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:46:10.676897 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:46:10.677986 24321 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } }
I20260619 18:46:10.678680 24321 leader_election.cc:302] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec2ebd58a99749f297489ff2ad54a291; no voters:
I20260619 18:46:10.679368 24321 leader_election.cc:288] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:10.679517 24331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:10.680631 24331 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [term 1 LEADER]: Becoming Leader. State: Replica: ec2ebd58a99749f297489ff2ad54a291, State: Running, Role: LEADER
I20260619 18:46:10.681327 24331 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } }
I20260619 18:46:10.681847 24321 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:10.684742 24333 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2ebd58a99749f297489ff2ad54a291. Latest consensus state: current_term: 1 leader_uuid: "ec2ebd58a99749f297489ff2ad54a291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } } }
I20260619 18:46:10.685396 24333 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:10.685225 24332 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec2ebd58a99749f297489ff2ad54a291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2ebd58a99749f297489ff2ad54a291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35645 } } }
I20260619 18:46:10.685756 24332 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:10.688725 24338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:10.695896 24338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:10.705710 24338 catalog_manager.cc:1347] Generated new cluster ID: cd56e6bfa4924f2b8b3d1468bcbac5ef
I20260619 18:46:10.705865 24338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:10.735776 24338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:10.737228 24338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:10.747083 24338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2ebd58a99749f297489ff2ad54a291: Generated new TSK 0
I20260619 18:46:10.747992 24338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:10.764289 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24254
W20260619 18:46:10.765846 24316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55471
W20260619 18:46:10.766836 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:10.767007 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24254
2026-06-19T18:46:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:10 dist-test-slave-09fp krb5kdc[24352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:10 dist-test-slave-09fp krb5kdc[24352](info): set up 2 sockets
Jun 19 18:46:10 dist-test-slave-09fp krb5kdc[24352](info): commencing operation
krb5kdc: starting...
W20260619 18:46:11.890988 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:12 dist-test-slave-09fp krb5kdc[24352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:12.178860 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:12.529819 24365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:12.530215 24365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:12.530392 24365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:12.538609 24365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:12.538832 24365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:12.538961 24365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:12.539086 24365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:12.539171 24365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:12.550608 24365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:12.552152 24365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:12.554064 24365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:12.567723 24370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:12.567735 24373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:12.567663 24371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:12.568261 24365 server_base.cc:1034] running on GCE node
I20260619 18:46:12.569312 24365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:12.571033 24365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:12.572299 24365 hybrid_clock.cc:648] HybridClock initialized: now 1781894772572249 us; error 33 us; skew 500 ppm
I20260619 18:46:12.572722 24365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:12 dist-test-slave-09fp krb5kdc[24352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:12.576500 24365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:12.578315 24365 webserver.cc:460] Webserver started at http://127.0.0.1:35943/ using document root <none> and password file <none>
I20260619 18:46:12.579171 24365 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:12.579322 24365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:12.579674 24365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:12.583628 24365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "8c96f3ae61224289b9465d121c49e3e4"
format_stamp: "Formatted at 2026-06-19 18:46:12 on dist-test-slave-09fp"
server_key: "720282dd16a5ac4e56ceb64cfb2f8d74"
server_key_iv: "c648343a82d14e8e5dac8dd866ef824a"
server_key_version: "encryptionkey@0"
I20260619 18:46:12.584527 24365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "8c96f3ae61224289b9465d121c49e3e4"
format_stamp: "Formatted at 2026-06-19 18:46:12 on dist-test-slave-09fp"
server_key: "720282dd16a5ac4e56ceb64cfb2f8d74"
server_key_iv: "c648343a82d14e8e5dac8dd866ef824a"
server_key_version: "encryptionkey@0"
I20260619 18:46:12.590425 24365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 18:46:12.594830 24380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:12.596371 24365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:46:12.596624 24365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "8c96f3ae61224289b9465d121c49e3e4"
format_stamp: "Formatted at 2026-06-19 18:46:12 on dist-test-slave-09fp"
server_key: "720282dd16a5ac4e56ceb64cfb2f8d74"
server_key_iv: "c648343a82d14e8e5dac8dd866ef824a"
server_key_version: "encryptionkey@0"
I20260619 18:46:12.596894 24365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:12.603498 24365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:12.608276 24365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:12.608615 24365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:12.628654 24365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43265
I20260619 18:46:12.628702 24431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43265 every 8 connection(s)
I20260619 18:46:12.630234 24365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:12.634588 24432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:12.635675 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24365
I20260619 18:46:12.635872 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:12.636457 3414 external_mini_cluster.cc:1428] Setting key 5828a8f73c8f86647ce49c66d105a75e
I20260619 18:46:12.645118 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:12.638403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42283 (local address 127.0.0.1:43265)
0619 18:46:12.639423 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:12.639436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:12.639487 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:46:12.641409 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:12.641438 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:12.641891 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:12.642274 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:12.643462 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:12.643469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:12.643483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:12.643499 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:12.643562 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:12.643587 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:12.643784 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:12.643890 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 18:46:12.644142 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":439,"threads_started":1}
I20260619 18:46:12.649008 24432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4: Bootstrap starting.
I20260619 18:46:12.649236 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:12.653430 24432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:12.655105 24432 log.cc:824] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:12.657987 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:12.654992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55855 (local address 127.0.0.1:43265)
0619 18:46:12.655239 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:12.655245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:12.655563 (+ 318us) server_negotiation.cc:407] Connection header received
0619 18:46:12.655814 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:12.655818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:12.655880 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:12.656034 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:12.657137 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:12.657142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:12.657145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:12.657165 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:12.657210 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:12.657229 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:12.657353 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:12.657570 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:46:12.657625 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 18:46:12.659142 24432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4: No bootstrap required, opened a new log
I20260619 18:46:12.667729 24432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } }
I20260619 18:46:12.668160 24432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:12.668237 24432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c96f3ae61224289b9465d121c49e3e4, State: Initialized, Role: FOLLOWER
I20260619 18:46:12.668895 24432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } }
I20260619 18:46:12.669126 24432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:12.669240 24432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:12.669373 24432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:12.671937 24432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } }
I20260619 18:46:12.672456 24432 leader_election.cc:302] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c96f3ae61224289b9465d121c49e3e4; no voters:
I20260619 18:46:12.673048 24432 leader_election.cc:288] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:12.673197 24442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:12.673926 24442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [term 1 LEADER]: Becoming Leader. State: Replica: 8c96f3ae61224289b9465d121c49e3e4, State: Running, Role: LEADER
I20260619 18:46:12.674551 24442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } }
I20260619 18:46:12.675405 24432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:12.677850 24444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c96f3ae61224289b9465d121c49e3e4. Latest consensus state: current_term: 1 leader_uuid: "8c96f3ae61224289b9465d121c49e3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } } }
I20260619 18:46:12.678289 24444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:12.678764 24443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c96f3ae61224289b9465d121c49e3e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c96f3ae61224289b9465d121c49e3e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43265 } } }
I20260619 18:46:12.679256 24443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:12.681452 24447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:12.687768 24447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:12.697568 24447 catalog_manager.cc:1347] Generated new cluster ID: 99124489b9aa4ba2a5988e16b2ca0b77
I20260619 18:46:12.697731 24447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:12.716761 24447 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:12.717414 24447 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:12.724960 24447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c96f3ae61224289b9465d121c49e3e4: Generated new TSK 0
I20260619 18:46:12.726120 24447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:12.744378 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24365
W20260619 18:46:12.745684 24427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42283
W20260619 18:46:12.746580 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:12.746837 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24365
2026-06-19T18:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:12 dist-test-slave-09fp krb5kdc[24463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:12 dist-test-slave-09fp krb5kdc[24463](info): set up 2 sockets
Jun 19 18:46:12 dist-test-slave-09fp krb5kdc[24463](info): commencing operation
krb5kdc: starting...
W20260619 18:46:13.863054 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:14.138548 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:14.483695 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:14.484098 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:14.484282 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:14.491838 24476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:14.492033 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:14.492149 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:14.492273 24476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:14.492363 24476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:14.502653 24476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:14.504179 24476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:14.505653 24476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:14.518641 24481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:14.518585 24484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:14.518626 24482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:14.519897 24476 server_base.cc:1034] running on GCE node
I20260619 18:46:14.520800 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:14.522508 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:14.523795 24476 hybrid_clock.cc:648] HybridClock initialized: now 1781894774523729 us; error 67 us; skew 500 ppm
I20260619 18:46:14.524205 24476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:14.527733 24476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:14.529487 24476 webserver.cc:460] Webserver started at http://127.0.0.1:37667/ using document root <none> and password file <none>
I20260619 18:46:14.530294 24476 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:14.530514 24476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:14.530853 24476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:14.534164 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "9d8311d2411247559bdd952e204f9579"
format_stamp: "Formatted at 2026-06-19 18:46:14 on dist-test-slave-09fp"
server_key: "8a47d5dc75813536a253ded9fbdb85a8"
server_key_iv: "a18d03cf6b508efb2fce7a88c4ac042d"
server_key_version: "encryptionkey@0"
I20260619 18:46:14.535164 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "9d8311d2411247559bdd952e204f9579"
format_stamp: "Formatted at 2026-06-19 18:46:14 on dist-test-slave-09fp"
server_key: "8a47d5dc75813536a253ded9fbdb85a8"
server_key_iv: "a18d03cf6b508efb2fce7a88c4ac042d"
server_key_version: "encryptionkey@0"
I20260619 18:46:14.540985 24476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:14.545168 24491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:14.546736 24476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:14.546957 24476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "9d8311d2411247559bdd952e204f9579"
format_stamp: "Formatted at 2026-06-19 18:46:14 on dist-test-slave-09fp"
server_key: "8a47d5dc75813536a253ded9fbdb85a8"
server_key_iv: "a18d03cf6b508efb2fce7a88c4ac042d"
server_key_version: "encryptionkey@0"
I20260619 18:46:14.547192 24476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:14.572055 24476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:14.576622 24476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:14.576943 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:14.596911 24476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34605
I20260619 18:46:14.596949 24542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34605 every 8 connection(s)
I20260619 18:46:14.598384 24476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:14.602270 24543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:14.606428 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24476
I20260619 18:46:14.606690 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:14.607105 3414 external_mini_cluster.cc:1428] Setting key a06dfff65fab1f1c8879f4f3d1f1af82
I20260619 18:46:14.613864 24543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579: Bootstrap starting.
I20260619 18:46:14.617210 24543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:14.618340 24543 log.cc:824] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:14.622212 24543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579: No bootstrap required, opened a new log
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:14.633519 24543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260619 18:46:14.634075 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:14.609113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:36859 (local address 127.0.0.1:34605)
0619 18:46:14.609900 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:14.609914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:14.609960 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:46:14.612505 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:14.612534 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:14.612981 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:14.613411 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:14.615064 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:14.616447 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:14.617882 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:14.618725 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:14.618945 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:14.624168 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:14.624185 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:14.624225 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:14.624288 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:14.628693 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:14.629758 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:14.629769 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:14.629779 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:14.630014 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:14.630536 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:14.630542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:14.630545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:14.631414 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:14.631806 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:14.632255 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 18:46:14.632651 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260619 18:46:14.634043 24543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:14.634212 24543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8311d2411247559bdd952e204f9579, State: Initialized, Role: FOLLOWER
I20260619 18:46:14.634931 24543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260619 18:46:14.635167 24543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:14.635273 24543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:14.635408 24543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:14.638737 24543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260619 18:46:14.638978 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:14.639379 24543 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8311d2411247559bdd952e204f9579; no voters:
I20260619 18:46:14.640053 24543 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:14.640182 24548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:14.641597 24548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [term 1 LEADER]: Becoming Leader. State: Replica: 9d8311d2411247559bdd952e204f9579, State: Running, Role: LEADER
I20260619 18:46:14.642171 24548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } }
I20260619 18:46:14.642486 24543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:14.645097 24550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8311d2411247559bdd952e204f9579. Latest consensus state: current_term: 1 leader_uuid: "9d8311d2411247559bdd952e204f9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } } }
I20260619 18:46:14.645536 24550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:14.645196 24549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8311d2411247559bdd952e204f9579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8311d2411247559bdd952e204f9579" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34605 } } }
I20260619 18:46:14.646191 24549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:14.649132 24558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:14.656579 24558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:14.670953 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:14.651335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:59277 (local address 127.0.0.1:34605)
0619 18:46:14.651524 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:14.651530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:14.651558 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:46:14.651643 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:14.651656 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:14.651750 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:14.651873 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:14.654217 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:14.655577 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:14.657783 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:14.658522 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:14.658640 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:14.662473 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:14.662481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:14.662486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:14.662538 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:14.666593 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:14.667896 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:14.667902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:14.667905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:14.668090 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:14.668706 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:14.668713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:14.668721 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:14.669197 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:14.669493 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:14.670298 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 18:46:14.670555 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:46:14.674512 24558 catalog_manager.cc:1347] Generated new cluster ID: ab53f4c41a134d7b85aaa294eb3048cc
I20260619 18:46:14.674636 24558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:14.693351 24558 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:14.695395 24558 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:14.717270 24558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8311d2411247559bdd952e204f9579: Generated new TSK 0
I20260619 18:46:14.718264 24558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:14.726369 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24476
I20260619 18:46:14.727914 24538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:14.822650 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24476
W20260619 18:46:14.840584 24461 connection.cc:535] client connection to 127.0.0.1:34605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34605): Connection reset by peer (error 104)
2026-06-19T18:46:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24575](info): set up 2 sockets
Jun 19 18:46:14 dist-test-slave-09fp krb5kdc[24575](info): commencing operation
krb5kdc: starting...
W20260619 18:46:15.951107 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:16 dist-test-slave-09fp krb5kdc[24575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:16.226857 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:16.575528 24588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:16.575855 24588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:16.576071 24588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:16.583524 24588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:16.583652 24588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:16.583783 24588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:16.583896 24588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:16.583982 24588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:16.594786 24588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:16.596237 24588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:16.597620 24588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:16.610785 24593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:16.610754 24596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:16.610739 24594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:16.612422 24588 server_base.cc:1034] running on GCE node
I20260619 18:46:16.613358 24588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:16.615186 24588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:16.616454 24588 hybrid_clock.cc:648] HybridClock initialized: now 1781894776616388 us; error 57 us; skew 500 ppm
I20260619 18:46:16.616868 24588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:16 dist-test-slave-09fp krb5kdc[24575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:16.624135 24588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:16.626065 24588 webserver.cc:460] Webserver started at http://127.0.0.1:45991/ using document root <none> and password file <none>
I20260619 18:46:16.626979 24588 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:16.627147 24588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:16.627523 24588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:16.631287 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "b757773f07b94d7c814cc00088603f41"
format_stamp: "Formatted at 2026-06-19 18:46:16 on dist-test-slave-09fp"
server_key: "5a2bc7b0d6ec6b713c9386b58edbe4eb"
server_key_iv: "fefd6a2877eda2f45ff04184f3fff224"
server_key_version: "encryptionkey@0"
I20260619 18:46:16.632290 24588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "b757773f07b94d7c814cc00088603f41"
format_stamp: "Formatted at 2026-06-19 18:46:16 on dist-test-slave-09fp"
server_key: "5a2bc7b0d6ec6b713c9386b58edbe4eb"
server_key_iv: "fefd6a2877eda2f45ff04184f3fff224"
server_key_version: "encryptionkey@0"
I20260619 18:46:16.638559 24588 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:46:16.643280 24603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:16.645048 24588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:46:16.645308 24588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "b757773f07b94d7c814cc00088603f41"
format_stamp: "Formatted at 2026-06-19 18:46:16 on dist-test-slave-09fp"
server_key: "5a2bc7b0d6ec6b713c9386b58edbe4eb"
server_key_iv: "fefd6a2877eda2f45ff04184f3fff224"
server_key_version: "encryptionkey@0"
I20260619 18:46:16.645568 24588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:16.661526 24588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:16.666177 24588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:16.666536 24588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:16.686764 24588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39573
I20260619 18:46:16.686805 24654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39573 every 8 connection(s)
I20260619 18:46:16.688271 24588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:16.692276 24655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:16.694406 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24588
I20260619 18:46:16.694636 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:16.695179 3414 external_mini_cluster.cc:1428] Setting key 7001ed9afcc6415b16b9ac9fa4f1cec1
I20260619 18:46:16.706954 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41: Bootstrap starting.
I20260619 18:46:16.711297 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:16.697358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:51975 (local address 127.0.0.1:39573)
0619 18:46:16.698232 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:16.698249 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:16.698292 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:46:16.700098 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:16.700130 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:16.700642 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:16.701065 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:16.701657 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:16.703066 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:16.704577 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:16.705351 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:16.705586 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:16.707155 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:16.707167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:16.707191 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:16.708150 (+ 959us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:16.708298 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:16.708338 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:16.708549 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:16.709052 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 18:46:16.709375 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":363,"threads_started":1}
I20260619 18:46:16.712396 24655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:16.713912 24655 log.cc:824] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:16.717861 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:16.717895 24655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41: No bootstrap required, opened a new log
I20260619 18:46:16.728024 24655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260619 18:46:16.728628 24655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:16.728768 24655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b757773f07b94d7c814cc00088603f41, State: Initialized, Role: FOLLOWER
I20260619 18:46:16.729537 24655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260619 18:46:16.729871 24655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:16.729969 24658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:16.722704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:54357 (local address 127.0.0.1:39573)
0619 18:46:16.722919 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:16.722925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:16.723276 (+ 351us) server_negotiation.cc:407] Connection header received
0619 18:46:16.723484 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:16.723489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:16.723585 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:16.723676 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:16.724342 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:16.725142 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:16.727097 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:16.727558 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:16.727660 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:16.729089 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:16.729094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:16.729098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:16.729118 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:16.729179 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:16.729195 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:16.729340 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:16.729564 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:46:16.729625 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:46:16.730046 24655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:16.730245 24655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:16.732980 24655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260619 18:46:16.733649 24655 leader_election.cc:302] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b757773f07b94d7c814cc00088603f41; no voters:
I20260619 18:46:16.734344 24655 leader_election.cc:288] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:16.734490 24665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:16.735242 24665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [term 1 LEADER]: Becoming Leader. State: Replica: b757773f07b94d7c814cc00088603f41, State: Running, Role: LEADER
I20260619 18:46:16.735913 24665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } }
I20260619 18:46:16.736822 24655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:16.739094 24666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b757773f07b94d7c814cc00088603f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } } }
I20260619 18:46:16.739609 24666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:16.739471 24667 sys_catalog.cc:455] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b757773f07b94d7c814cc00088603f41. Latest consensus state: current_term: 1 leader_uuid: "b757773f07b94d7c814cc00088603f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b757773f07b94d7c814cc00088603f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39573 } } }
I20260619 18:46:16.739917 24667 sys_catalog.cc:458] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:16.743952 24671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:16.750406 24671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:16.760537 24671 catalog_manager.cc:1347] Generated new cluster ID: 15553f0b2f2941cc9f3880ea5d602f79
I20260619 18:46:16.760690 24671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:16.773272 24671 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:16.774875 24671 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:16.790107 24671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b757773f07b94d7c814cc00088603f41: Generated new TSK 0
I20260619 18:46:16.791123 24671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:16.802335 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24588
W20260619 18:46:16.803907 24650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:51975
W20260619 18:46:16.804827 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:16.804986 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24588
W20260619 18:46:16.822404 24573 connection.cc:535] client connection to 127.0.0.1:39573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39573): Connection reset by peer (error 104)
2026-06-19T18:46:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:16 dist-test-slave-09fp krb5kdc[24686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:16 dist-test-slave-09fp krb5kdc[24686](info): set up 2 sockets
Jun 19 18:46:16 dist-test-slave-09fp krb5kdc[24686](info): commencing operation
krb5kdc: starting...
W20260619 18:46:17.939499 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:18 dist-test-slave-09fp krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:18.217655 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:18.568842 24699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:18.569216 24699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:18.569397 24699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:18.577082 24699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:18.577271 24699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:18.577389 24699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:18.577508 24699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:18.577598 24699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:18.588124 24699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:18.589607 24699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:18.591085 24699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:18.603533 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:18.604240 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:18.604465 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:18.605110 24699 server_base.cc:1034] running on GCE node
I20260619 18:46:18.606078 24699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:18.607827 24699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:18.609076 24699 hybrid_clock.cc:648] HybridClock initialized: now 1781894778609029 us; error 37 us; skew 500 ppm
I20260619 18:46:18.609515 24699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:18 dist-test-slave-09fp krb5kdc[24686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:18.613154 24699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:18.615049 24699 webserver.cc:460] Webserver started at http://127.0.0.1:42951/ using document root <none> and password file <none>
I20260619 18:46:18.615877 24699 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:18.616035 24699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:18.616407 24699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:18.619885 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "5ea95365231b4ffd84fa2ab5584106df"
format_stamp: "Formatted at 2026-06-19 18:46:18 on dist-test-slave-09fp"
server_key: "06e6c9a41cf0e2152f6593c803934fc6"
server_key_iv: "7342408975c9fdeb64d1258a4f9d24ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:18.620827 24699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "5ea95365231b4ffd84fa2ab5584106df"
format_stamp: "Formatted at 2026-06-19 18:46:18 on dist-test-slave-09fp"
server_key: "06e6c9a41cf0e2152f6593c803934fc6"
server_key_iv: "7342408975c9fdeb64d1258a4f9d24ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:18.626943 24699 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 18:46:18.630864 24714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:18.632390 24699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:18.632617 24699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "5ea95365231b4ffd84fa2ab5584106df"
format_stamp: "Formatted at 2026-06-19 18:46:18 on dist-test-slave-09fp"
server_key: "06e6c9a41cf0e2152f6593c803934fc6"
server_key_iv: "7342408975c9fdeb64d1258a4f9d24ae"
server_key_version: "encryptionkey@0"
I20260619 18:46:18.632856 24699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:18.639246 24699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:18.643610 24699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:18.643952 24699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:18.664019 24699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33689
I20260619 18:46:18.664064 24765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33689 every 8 connection(s)
I20260619 18:46:18.665657 24699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:18.669797 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:18.675432 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24699
I20260619 18:46:18.675621 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:18.676146 3414 external_mini_cluster.cc:1428] Setting key 2ccce38e36dac83f054fb9e229b965ec
I20260619 18:46:18.682073 24766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df: Bootstrap starting.
I20260619 18:46:18.684705 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:18.678171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:53293 (local address 127.0.0.1:33689)
0619 18:46:18.679044 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:18.679057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:18.679102 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:46:18.680970 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:18.680998 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:18.681400 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:18.681828 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:18.682983 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:18.682989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:18.683003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:18.683018 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:18.683086 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:18.683108 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:18.683365 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:18.683516 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:46:18.683728 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":387,"threads_started":1}
I20260619 18:46:18.685724 24766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:18.686976 24766 log.cc:824] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:18.688973 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:18.690299 24766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df: No bootstrap required, opened a new log
I20260619 18:46:18.696282 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:18.693532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:56607 (local address 127.0.0.1:33689)
0619 18:46:18.693780 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:18.693790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:18.694323 (+ 533us) server_negotiation.cc:407] Connection header received
0619 18:46:18.694479 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:18.694485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:18.694552 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:18.694671 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:18.695597 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:18.695605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:18.695615 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:18.695635 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:18.695688 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:18.695713 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:18.695853 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:18.695897 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 18:46:18.695958 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:46:18.699846 24766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } }
I20260619 18:46:18.700268 24766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:18.700392 24766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea95365231b4ffd84fa2ab5584106df, State: Initialized, Role: FOLLOWER
I20260619 18:46:18.701010 24766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } }
I20260619 18:46:18.701270 24766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:18.701438 24766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:18.701620 24766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:18.704087 24766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } }
I20260619 18:46:18.704744 24766 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea95365231b4ffd84fa2ab5584106df; no voters:
I20260619 18:46:18.705328 24766 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:18.705488 24776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:18.711308 24776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [term 1 LEADER]: Becoming Leader. State: Replica: 5ea95365231b4ffd84fa2ab5584106df, State: Running, Role: LEADER
I20260619 18:46:18.711772 24766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:18.712080 24776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } }
I20260619 18:46:18.723549 24778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea95365231b4ffd84fa2ab5584106df. Latest consensus state: current_term: 1 leader_uuid: "5ea95365231b4ffd84fa2ab5584106df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } } }
I20260619 18:46:18.723590 24777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea95365231b4ffd84fa2ab5584106df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea95365231b4ffd84fa2ab5584106df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33689 } } }
I20260619 18:46:18.724119 24778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:18.724339 24777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:18.729045 24791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:18.734115 24791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:18.744150 24791 catalog_manager.cc:1347] Generated new cluster ID: 90ffdcf3111549fcbe886eb1308b8c5a
I20260619 18:46:18.744347 24791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:18.766080 24791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:18.766693 24791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:18.777514 24791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea95365231b4ffd84fa2ab5584106df: Generated new TSK 0
I20260619 18:46:18.778501 24791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:18.787750 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24699
W20260619 18:46:18.789175 24761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:53293
W20260619 18:46:18.790043 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24699: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:18.790221 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24699
2026-06-19T18:46:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:18 dist-test-slave-09fp krb5kdc[24797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:18 dist-test-slave-09fp krb5kdc[24797](info): set up 2 sockets
Jun 19 18:46:18 dist-test-slave-09fp krb5kdc[24797](info): commencing operation
krb5kdc: starting...
W20260619 18:46:19.915030 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:20.195621 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:20.544270 24810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:20.544592 24810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:20.544766 24810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:20.552326 24810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:20.552457 24810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:20.552583 24810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:20.552708 24810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:20.552779 24810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:20.563146 24810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:20.564610 24810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:20.566035 24810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:20.579175 24815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:20.579156 24818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:20.579413 24816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:20.580111 24810 server_base.cc:1034] running on GCE node
I20260619 18:46:20.581043 24810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:20.582707 24810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:20.583974 24810 hybrid_clock.cc:648] HybridClock initialized: now 1781894780583924 us; error 46 us; skew 500 ppm
I20260619 18:46:20.584372 24810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:20.587936 24810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:20.589658 24810 webserver.cc:460] Webserver started at http://127.0.0.1:42449/ using document root <none> and password file <none>
I20260619 18:46:20.590417 24810 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:20.590642 24810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:20.591065 24810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:20.594897 24810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "b87c2d3b97624bc2b5444bb8104ae06f"
format_stamp: "Formatted at 2026-06-19 18:46:20 on dist-test-slave-09fp"
server_key: "a85968568fb119791ee880f7ab8794fd"
server_key_iv: "a93be20c13c6c53175584d0b4c608ec9"
server_key_version: "encryptionkey@0"
I20260619 18:46:20.595767 24810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "b87c2d3b97624bc2b5444bb8104ae06f"
format_stamp: "Formatted at 2026-06-19 18:46:20 on dist-test-slave-09fp"
server_key: "a85968568fb119791ee880f7ab8794fd"
server_key_iv: "a93be20c13c6c53175584d0b4c608ec9"
server_key_version: "encryptionkey@0"
I20260619 18:46:20.601410 24810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 18:46:20.605254 24825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:20.606814 24810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:20.607023 24810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "b87c2d3b97624bc2b5444bb8104ae06f"
format_stamp: "Formatted at 2026-06-19 18:46:20 on dist-test-slave-09fp"
server_key: "a85968568fb119791ee880f7ab8794fd"
server_key_iv: "a93be20c13c6c53175584d0b4c608ec9"
server_key_version: "encryptionkey@0"
I20260619 18:46:20.607247 24810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:20.625419 24810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:20.630515 24810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:20.630864 24810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:20.650291 24810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43583
I20260619 18:46:20.650332 24876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43583 every 8 connection(s)
I20260619 18:46:20.651903 24810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:20.653642 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24810
I20260619 18:46:20.653851 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:20.654306 3414 external_mini_cluster.cc:1428] Setting key 8273427ca59b335334c2aadd81adbed7
I20260619 18:46:20.656788 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:46:20.671397 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f: Bootstrap starting.
I20260619 18:46:20.676218 24877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:20.677596 24877 log.cc:824] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:20.681537 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f: No bootstrap required, opened a new log
I20260619 18:46:20.682021 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:20.656346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:35365 (local address 127.0.0.1:43583)
0619 18:46:20.657225 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:20.657241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:20.657295 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:46:20.659718 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:20.659748 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:20.660391 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:20.660909 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:20.662368 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:20.664591 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:20.666060 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:20.667235 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:20.667455 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:20.672374 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:20.672387 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:20.672415 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:20.672481 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:20.676812 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:20.677837 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:20.677846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:20.677855 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:20.678085 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:20.678742 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:20.678746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:20.678748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:20.679433 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:20.679746 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:20.680251 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 18:46:20.680621 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":361,"threads_started":1}
I20260619 18:46:20.686658 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:20.690703 24877 raft_consensus.cc:348] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } }
I20260619 18:46:20.691304 24877 raft_consensus.cc:374] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:20.691491 24877 raft_consensus.cc:729] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87c2d3b97624bc2b5444bb8104ae06f, State: Initialized, Role: FOLLOWER
I20260619 18:46:20.692201 24877 consensus_queue.cc:260] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } }
I20260619 18:46:20.692517 24877 raft_consensus.cc:388] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:20.692677 24877 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:20.692859 24877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:20.695307 24877 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } }
I20260619 18:46:20.695889 24877 leader_election.cc:302] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87c2d3b97624bc2b5444bb8104ae06f; no voters:
I20260619 18:46:20.696625 24877 leader_election.cc:288] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:20.696805 24887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:20.698055 24887 raft_consensus.cc:686] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [term 1 LEADER]: Becoming Leader. State: Replica: b87c2d3b97624bc2b5444bb8104ae06f, State: Running, Role: LEADER
I20260619 18:46:20.698830 24887 consensus_queue.cc:237] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } }
I20260619 18:46:20.699419 24877 sys_catalog.cc:564] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:20.702167 24888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } } }
I20260619 18:46:20.702750 24888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:20.710819 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:20.691234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:54905 (local address 127.0.0.1:43583)
0619 18:46:20.691449 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:20.691455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:20.691890 (+ 435us) server_negotiation.cc:407] Connection header received
0619 18:46:20.692100 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:20.692106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:20.692191 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:20.692302 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:20.693975 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:20.694891 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:20.696455 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:20.697352 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:20.698361 (+ 1009us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:20.700508 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:20.700518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:20.700524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:20.700581 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:20.704711 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:20.706231 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:20.706240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:20.706247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:20.706543 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:20.707308 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:20.707314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:20.707321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:20.707814 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:20.709574 (+ 1760us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:20.709826 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:46:20.710132 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:46:20.713598 24889 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b87c2d3b97624bc2b5444bb8104ae06f. Latest consensus state: current_term: 1 leader_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87c2d3b97624bc2b5444bb8104ae06f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43583 } } }
I20260619 18:46:20.714614 24889 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:20.715572 24892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:20.722002 24892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:20.732720 24892 catalog_manager.cc:1347] Generated new cluster ID: 383fb9db04e2428c9d97522becaa97c2
I20260619 18:46:20.732856 24892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:20.752295 24892 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:20.753717 24892 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:20.763898 24892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b87c2d3b97624bc2b5444bb8104ae06f: Generated new TSK 0
I20260619 18:46:20.764843 24892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:20.773890 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24810
I20260619 18:46:20.775278 24872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:20.868271 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24810
W20260619 18:46:20.886725 24795 connection.cc:535] client connection to 127.0.0.1:43583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43583): Connection reset by peer (error 104)
2026-06-19T18:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24909](info): set up 2 sockets
Jun 19 18:46:20 dist-test-slave-09fp krb5kdc[24909](info): commencing operation
krb5kdc: starting...
W20260619 18:46:21.995065 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:22 dist-test-slave-09fp krb5kdc[24909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:22.299161 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:22.661073 24922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:22.661474 24922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:22.661654 24922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:22.669368 24922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:22.669570 24922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:22.669683 24922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:22.669800 24922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:22.669893 24922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:22.680652 24922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:22.682176 24922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:22.683677 24922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:22.696585 24927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.696496 24928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:22.696511 24930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:22.697937 24922 server_base.cc:1034] running on GCE node
I20260619 18:46:22.698904 24922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:22.700501 24922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:22.701779 24922 hybrid_clock.cc:648] HybridClock initialized: now 1781894782701733 us; error 42 us; skew 500 ppm
I20260619 18:46:22.702183 24922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:22 dist-test-slave-09fp krb5kdc[24909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:22.705946 24922 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:22.708441 24922 webserver.cc:460] Webserver started at http://127.0.0.1:44121/ using document root <none> and password file <none>
I20260619 18:46:22.709684 24922 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:22.709861 24922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:22.710286 24922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:22.714591 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "979f45556d3b4cf28a06995c9917d089"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-09fp"
server_key: "aaa1f61a82015707f3463de60cf12609"
server_key_iv: "ddf7bc4cccea0ddbe3b59fc6cdbf65fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.715503 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "979f45556d3b4cf28a06995c9917d089"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-09fp"
server_key: "aaa1f61a82015707f3463de60cf12609"
server_key_iv: "ddf7bc4cccea0ddbe3b59fc6cdbf65fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.721719 24922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:46:22.726292 24937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:22.727974 24922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:46:22.728226 24922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "979f45556d3b4cf28a06995c9917d089"
format_stamp: "Formatted at 2026-06-19 18:46:22 on dist-test-slave-09fp"
server_key: "aaa1f61a82015707f3463de60cf12609"
server_key_iv: "ddf7bc4cccea0ddbe3b59fc6cdbf65fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:22.728452 24922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:22.750237 24922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:22.755419 24922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:22.755769 24922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:22.775949 24922 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46145
I20260619 18:46:22.775970 24988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46145 every 8 connection(s)
I20260619 18:46:22.777462 24922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:22.781662 24989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:22.787570 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 24922
I20260619 18:46:22.787724 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:22.788196 3414 external_mini_cluster.cc:1428] Setting key 808bdc30a82b7d2dd96c17cc26db0c23
I20260619 18:46:22.794610 24989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089: Bootstrap starting.
I20260619 18:46:22.799027 24989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:22.800642 24989 log.cc:824] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:22.804068 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.790317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:57233 (local address 127.0.0.1:46145)
0619 18:46:22.791319 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.791333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.791377 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:46:22.793217 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.793244 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.793715 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.794102 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.794935 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.796300 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.798560 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.799453 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.799642 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.801153 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.801163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.801179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:22.802066 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.802179 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:22.802216 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.802383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.802821 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.803046 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":380,"threads_started":1}
I20260619 18:46:22.804438 24989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089: No bootstrap required, opened a new log
I20260619 18:46:22.808429 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:22.814072 24989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260619 18:46:22.814697 24989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:22.814863 24989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979f45556d3b4cf28a06995c9917d089, State: Initialized, Role: FOLLOWER
I20260619 18:46:22.815699 24989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260619 18:46:22.816080 24989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:22.816277 24989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:22.816495 24989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:22.820082 24989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260619 18:46:22.820896 24989 leader_election.cc:302] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 979f45556d3b4cf28a06995c9917d089; no voters:
I20260619 18:46:22.821833 24989 leader_election.cc:288] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:22.822078 24999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:22.822553 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:22.813521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:38999 (local address 127.0.0.1:46145)
0619 18:46:22.813852 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:22.813862 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:22.814657 (+ 795us) server_negotiation.cc:407] Connection header received
0619 18:46:22.814880 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:22.814886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:22.814966 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:22.815084 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:22.815805 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.816828 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.819037 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:22.819530 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:22.819637 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:22.821241 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:22.821247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:22.821251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:22.821280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:22.821357 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:22.821377 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:22.821481 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:22.821841 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 18:46:22.821908 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:46:22.823946 24999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [term 1 LEADER]: Becoming Leader. State: Replica: 979f45556d3b4cf28a06995c9917d089, State: Running, Role: LEADER
I20260619 18:46:22.824645 24999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } }
I20260619 18:46:22.825001 24989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:22.827730 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "979f45556d3b4cf28a06995c9917d089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } } }
I20260619 18:46:22.828148 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:22.829185 25007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:22.828962 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 979f45556d3b4cf28a06995c9917d089. Latest consensus state: current_term: 1 leader_uuid: "979f45556d3b4cf28a06995c9917d089" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "979f45556d3b4cf28a06995c9917d089" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46145 } } }
I20260619 18:46:22.831804 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:22.838340 25007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:22.862002 25007 catalog_manager.cc:1347] Generated new cluster ID: 78d41ca4023e4676b61265fd70789dfe
I20260619 18:46:22.862190 25007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:22.882742 25007 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:22.884291 25007 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:22.894398 25007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 979f45556d3b4cf28a06995c9917d089: Generated new TSK 0
I20260619 18:46:22.895432 25007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:22.908322 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24922
W20260619 18:46:22.909782 24984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:57233
W20260619 18:46:22.910719 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 24922: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:22.910907 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 24922
W20260619 18:46:22.928764 24907 connection.cc:535] client connection to 127.0.0.1:46145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46145): Connection reset by peer (error 104)
2026-06-19T18:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:23 dist-test-slave-09fp krb5kdc[25020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:23 dist-test-slave-09fp krb5kdc[25020](info): set up 2 sockets
Jun 19 18:46:23 dist-test-slave-09fp krb5kdc[25020](info): commencing operation
krb5kdc: starting...
W20260619 18:46:24.035079 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:24 dist-test-slave-09fp krb5kdc[25020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:46:24.330176 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:24.692780 25033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:24.693190 25033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:24.693372 25033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:24.700994 25033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:24.701196 25033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:24.701310 25033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:24.701437 25033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:24.701530 25033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:24.718703 25033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:24.720314 25033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:24.721753 25033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:24.737203 25039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:24.737483 25041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:24.737201 25038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:24.737984 25033 server_base.cc:1034] running on GCE node
I20260619 18:46:24.739010 25033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:24.740646 25033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:24.741935 25033 hybrid_clock.cc:648] HybridClock initialized: now 1781894784741874 us; error 39 us; skew 500 ppm
I20260619 18:46:24.742337 25033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:24 dist-test-slave-09fp krb5kdc[25020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:24.746155 25033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:46:24.748287 25033 webserver.cc:460] Webserver started at http://127.0.0.1:38255/ using document root <none> and password file <none>
I20260619 18:46:24.749048 25033 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:24.749166 25033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:24.749557 25033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:24.753237 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "a390fafa3e6f4961836d0ba8e6965f01"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-09fp"
server_key: "44829017616d109703112ba0be363048"
server_key_iv: "f090cdfcc4a0f87d224ee0c38a0afa8e"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.754276 25033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "a390fafa3e6f4961836d0ba8e6965f01"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-09fp"
server_key: "44829017616d109703112ba0be363048"
server_key_iv: "f090cdfcc4a0f87d224ee0c38a0afa8e"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.760998 25033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:24.765228 25048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:24.766816 25033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:46:24.767038 25033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "a390fafa3e6f4961836d0ba8e6965f01"
format_stamp: "Formatted at 2026-06-19 18:46:24 on dist-test-slave-09fp"
server_key: "44829017616d109703112ba0be363048"
server_key_iv: "f090cdfcc4a0f87d224ee0c38a0afa8e"
server_key_version: "encryptionkey@0"
I20260619 18:46:24.767269 25033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:24.773959 25033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:24.778680 25033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:24.779026 25033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:24.801075 25033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32927
I20260619 18:46:24.801124 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32927 every 8 connection(s)
I20260619 18:46:24.802764 25033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:24.806871 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:24.809155 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25033
I20260619 18:46:24.809358 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:24.809908 3414 external_mini_cluster.cc:1428] Setting key 6ea8ba3d4b473abd293b018a941c1a62
I20260619 18:46:24.818915 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.812017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:59739 (local address 127.0.0.1:32927)
0619 18:46:24.812943 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.812957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.813005 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:46:24.814835 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.814864 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.815316 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.815798 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:24.817005 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:24.817012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:24.817027 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:24.817043 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:24.817110 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:24.817132 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:24.817395 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.817545 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.817759 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":422,"threads_started":1}
I20260619 18:46:24.821595 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01: Bootstrap starting.
I20260619 18:46:24.823190 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:24.825178 25100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:24.826681 25100 log.cc:824] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:24.830245 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01: No bootstrap required, opened a new log
I20260619 18:46:24.830623 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:24.827878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:51925 (local address 127.0.0.1:32927)
0619 18:46:24.828098 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:24.828106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:24.828325 (+ 219us) server_negotiation.cc:407] Connection header received
0619 18:46:24.828557 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:24.828561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:24.828618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:24.828731 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:24.829757 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:24.829761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:24.829765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:24.829782 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:24.829826 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:24.829842 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:24.829995 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:24.830078 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 18:46:24.830149 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:46:24.838903 25100 raft_consensus.cc:348] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 18:46:24.839354 25100 raft_consensus.cc:374] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:24.839463 25100 raft_consensus.cc:729] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a390fafa3e6f4961836d0ba8e6965f01, State: Initialized, Role: FOLLOWER
I20260619 18:46:24.840080 25100 consensus_queue.cc:260] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 18:46:24.840343 25100 raft_consensus.cc:388] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:24.840495 25100 raft_consensus.cc:482] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:24.840691 25100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:24.843276 25100 raft_consensus.cc:504] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 18:46:24.843808 25100 leader_election.cc:302] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a390fafa3e6f4961836d0ba8e6965f01; no voters:
I20260619 18:46:24.844417 25100 leader_election.cc:288] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:24.844617 25110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:24.845304 25110 raft_consensus.cc:686] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [term 1 LEADER]: Becoming Leader. State: Replica: a390fafa3e6f4961836d0ba8e6965f01, State: Running, Role: LEADER
I20260619 18:46:24.845887 25110 consensus_queue.cc:237] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } }
I20260619 18:46:24.846645 25100 sys_catalog.cc:564] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:24.849366 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a390fafa3e6f4961836d0ba8e6965f01. Latest consensus state: current_term: 1 leader_uuid: "a390fafa3e6f4961836d0ba8e6965f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260619 18:46:24.849786 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:24.850590 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a390fafa3e6f4961836d0ba8e6965f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a390fafa3e6f4961836d0ba8e6965f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32927 } } }
I20260619 18:46:24.850982 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:24.851977 25117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:24.857821 25117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:24.869038 25117 catalog_manager.cc:1347] Generated new cluster ID: 1b83e32615e046daac144b245e2a27db
I20260619 18:46:24.869196 25117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:24.908078 25117 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:24.908645 25117 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:24.918066 25117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a390fafa3e6f4961836d0ba8e6965f01: Generated new TSK 0
I20260619 18:46:24.919064 25117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:24.929140 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25033
W20260619 18:46:24.930619 25095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:59739
W20260619 18:46:24.931458 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25033: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:24.931635 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25033
2026-06-19T18:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:25 dist-test-slave-09fp krb5kdc[25131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:25 dist-test-slave-09fp krb5kdc[25131](info): set up 2 sockets
Jun 19 18:46:25 dist-test-slave-09fp krb5kdc[25131](info): commencing operation
krb5kdc: starting...
W20260619 18:46:26.087214 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:26 dist-test-slave-09fp krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:26.365605 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44765
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:44765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:26.734768 25144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:26.735102 25144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:26.735206 25144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:26.743093 25144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:26.743247 25144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:26.743302 25144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:26.743366 25144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:26.743418 25144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:26.754285 25144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:44765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:26.755908 25144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:26.757344 25144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:26.770802 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.770910 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:26.770938 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:26.771560 25144 server_base.cc:1034] running on GCE node
I20260619 18:46:26.772714 25144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:26.774390 25144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:26.775738 25144 hybrid_clock.cc:648] HybridClock initialized: now 1781894786775692 us; error 45 us; skew 500 ppm
I20260619 18:46:26.776180 25144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:26 dist-test-slave-09fp krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:26.779736 25144 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:26.781427 25144 webserver.cc:460] Webserver started at http://127.3.85.190:44057/ using document root <none> and password file <none>
I20260619 18:46:26.782205 25144 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:26.782370 25144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:26.782723 25144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:26.786289 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "da064299f017469a8441821abc834473"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-09fp"
server_key: "96c56ff454596c4f577456ea3f3ed2f6"
server_key_iv: "a6621ec3572a8c6d423d70e43badd5f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.787273 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "da064299f017469a8441821abc834473"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-09fp"
server_key: "96c56ff454596c4f577456ea3f3ed2f6"
server_key_iv: "a6621ec3572a8c6d423d70e43badd5f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.793334 25144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 18:46:26.797578 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:26.799232 25144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:26.799460 25144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "da064299f017469a8441821abc834473"
format_stamp: "Formatted at 2026-06-19 18:46:26 on dist-test-slave-09fp"
server_key: "96c56ff454596c4f577456ea3f3ed2f6"
server_key_iv: "a6621ec3572a8c6d423d70e43badd5f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:26.799702 25144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:26.813446 25144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:26.818233 25144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:26.818630 25144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:26.838783 25144 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:44765
I20260619 18:46:26.838820 25210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:44765 every 8 connection(s)
I20260619 18:46:26.840544 25144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:26.844728 25211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:26.846801 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25144
I20260619 18:46:26.847016 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:26.847487 3414 external_mini_cluster.cc:1428] Setting key bcef45de7e7346657d5e7cc01514f8dc
I20260619 18:46:26.858202 25211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: Bootstrap starting.
I20260619 18:46:26.862728 25211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: Neither blocks nor log segments found. Creating new log.
Jun 19 18:46:26 dist-test-slave-09fp krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:26.864054 25211 log.cc:824] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:26.867930 25211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: No bootstrap required, opened a new log
I20260619 18:46:26.874739 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.849564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56409 (local address 127.3.85.190:44765)
0619 18:46:26.850526 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.850539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.850584 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:46:26.852359 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.852384 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.852821 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.853202 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.855247 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.856688 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.858704 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.859512 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.859677 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.864957 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.864967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.864985 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.865028 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.869154 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.870222 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.870230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.870242 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.870750 (+ 508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.871374 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.871378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.871380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.872075 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.872461 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.873083 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.873448 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":425,"threads_started":1}
I20260619 18:46:26.877319 25211 raft_consensus.cc:348] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } }
I20260619 18:46:26.877892 25211 raft_consensus.cc:374] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:26.878022 25211 raft_consensus.cc:729] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da064299f017469a8441821abc834473, State: Initialized, Role: FOLLOWER
I20260619 18:46:26.878752 25211 consensus_queue.cc:260] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } }
I20260619 18:46:26.879066 25211 raft_consensus.cc:388] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:26.879222 25211 raft_consensus.cc:482] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:26.879364 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:26.879431 25211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:26.881815 25211 raft_consensus.cc:504] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } }
I20260619 18:46:26.882592 25211 leader_election.cc:302] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da064299f017469a8441821abc834473; no voters:
I20260619 18:46:26.883266 25211 leader_election.cc:288] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:26.883445 25220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:26.885984 25211 sys_catalog.cc:564] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:26.890480 25220 raft_consensus.cc:686] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [term 1 LEADER]: Becoming Leader. State: Replica: da064299f017469a8441821abc834473, State: Running, Role: LEADER
I20260619 18:46:26.891501 25220 consensus_queue.cc:237] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } }
I20260619 18:46:26.896762 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da064299f017469a8441821abc834473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } } }
I20260619 18:46:26.897259 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:26.897823 25222 sys_catalog.cc:455] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da064299f017469a8441821abc834473. Latest consensus state: current_term: 1 leader_uuid: "da064299f017469a8441821abc834473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da064299f017469a8441821abc834473" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44765 } } }
I20260619 18:46:26.898149 25222 sys_catalog.cc:458] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473 [sys.catalog]: This master's current role is: LEADER
W20260619 18:46:26.906121 25238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 18:46:26.906219 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:26.884427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56701 (local address 127.3.85.190:44765)
0619 18:46:26.884622 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:26.884627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:26.885804 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 18:46:26.886149 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:26.886157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:26.886260 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:26.886391 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:26.888997 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.890530 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.892565 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:26.893288 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:26.893498 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:26.897860 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:26.897867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:26.897871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:26.897928 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:26.902135 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.903444 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.903450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.903453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.903633 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:26.904253 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:26.904259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:26.904262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:26.904783 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:26.905113 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:26.905577 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 18:46:26.905820 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 18:46:26.906309 25238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:26.906502 25236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:26.908419 25236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:26.918017 25236 catalog_manager.cc:1347] Generated new cluster ID: 962f85c6aa8f4e0e9c4016d42f384e2b
I20260619 18:46:26.918181 25236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:26.933530 25236 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:26.935094 25236 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:26.947402 25236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da064299f017469a8441821abc834473: Generated new TSK 0
I20260619 18:46:26.948642 25236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:26.960079 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25144
I20260619 18:46:26.961751 25206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:27.060235 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25144
W20260619 18:46:27.078688 25129 connection.cc:535] client connection to 127.3.85.190:44765 recv error: Network error: failed to read from TLS socket (remote: 127.3.85.190:44765): Connection reset by peer (error 104)
2026-06-19T18:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:27 dist-test-slave-09fp krb5kdc[25244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:27 dist-test-slave-09fp krb5kdc[25244](info): set up 2 sockets
Jun 19 18:46:27 dist-test-slave-09fp krb5kdc[25244](info): commencing operation
krb5kdc: starting...
W20260619 18:46:28.199064 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:28 dist-test-slave-09fp krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:28.481911 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35161
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:45245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:28.841498 25257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:28.841836 25257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:28.842000 25257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:28.849596 25257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:28.849726 25257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:28.849858 25257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:28.849977 25257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:28.850059 25257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:28.861127 25257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:45245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:35161
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:28.862725 25257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:28.864244 25257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:28.877805 25263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:28.877895 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:28.877869 25262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:28.878906 25257 server_base.cc:1034] running on GCE node
I20260619 18:46:28.879927 25257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:28.881656 25257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:28.882911 25257 hybrid_clock.cc:648] HybridClock initialized: now 1781894788882857 us; error 51 us; skew 500 ppm
I20260619 18:46:28.883337 25257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:28 dist-test-slave-09fp krb5kdc[25244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894788, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:28.892997 25257 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:28.894989 25257 webserver.cc:460] Webserver started at http://127.3.85.190:33279/ using document root <none> and password file <none>
I20260619 18:46:28.895807 25257 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:28.895963 25257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:28.896315 25257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:28.900292 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-09fp"
server_key: "d4a986f23220954df8d956e4c30bc0f9"
server_key_iv: "e2f1d4e0140dc2c5544f1e8ab83a4ae8"
server_key_version: "encryptionkey@0"
I20260619 18:46:28.901199 25257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-09fp"
server_key: "d4a986f23220954df8d956e4c30bc0f9"
server_key_iv: "e2f1d4e0140dc2c5544f1e8ab83a4ae8"
server_key_version: "encryptionkey@0"
I20260619 18:46:28.907115 25257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 18:46:28.911157 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:28.912693 25257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:28.912945 25257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4"
format_stamp: "Formatted at 2026-06-19 18:46:28 on dist-test-slave-09fp"
server_key: "d4a986f23220954df8d956e4c30bc0f9"
server_key_iv: "e2f1d4e0140dc2c5544f1e8ab83a4ae8"
server_key_version: "encryptionkey@0"
I20260619 18:46:28.913187 25257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:28.929865 25257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:28.935011 25257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:28.935382 25257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:28.956344 25257 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:35161
I20260619 18:46:28.956365 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:35161 every 8 connection(s)
I20260619 18:46:28.957908 25257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:28.960569 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25257
I20260619 18:46:28.960767 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:28.961318 3414 external_mini_cluster.cc:1428] Setting key fe83acd8180abf67d2f37ccee921ead3
I20260619 18:46:28.962872 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:28.976514 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:28.963376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58963 (local address 127.3.85.190:35161)
0619 18:46:28.964248 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:28.964260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:28.964302 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:46:28.965964 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:28.965989 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:28.966521 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:28.966924 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:28.967587 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:28.969062 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:28.971144 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:28.971952 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:28.972117 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:28.973572 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:28.973582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:28.973599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:28.974320 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:28.974426 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:28.974538 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:28.974742 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:28.975162 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 18:46:28.975399 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":416,"threads_started":1}
I20260619 18:46:28.977711 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4: Bootstrap starting.
I20260619 18:46:28.981166 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:28.981653 25324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:28.982877 25324 log.cc:824] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:28.986974 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4: No bootstrap required, opened a new log
I20260619 18:46:28.993781 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:28.985740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33267 (local address 127.3.85.190:35161)
0619 18:46:28.985959 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:28.985968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:28.986588 (+ 620us) server_negotiation.cc:407] Connection header received
0619 18:46:28.986876 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:28.986883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:28.986983 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:28.987087 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:28.987951 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:28.989270 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:28.990818 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:28.991457 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:28.991640 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:28.992947 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:28.992953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:28.992957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:28.992979 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:28.993024 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:28.993044 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:28.993188 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:28.993391 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 18:46:28.993461 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:46:28.996666 25324 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } }
I20260619 18:46:28.997169 25324 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:28.997310 25324 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd59ae30e2e4bb8a24b9fd1ad96eba4, State: Initialized, Role: FOLLOWER
I20260619 18:46:28.997941 25324 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } }
I20260619 18:46:28.998200 25324 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:28.998380 25324 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:28.998631 25324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:29.001132 25324 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } }
I20260619 18:46:29.001824 25324 leader_election.cc:302] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd59ae30e2e4bb8a24b9fd1ad96eba4; no voters:
I20260619 18:46:29.002549 25324 leader_election.cc:288] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:29.002700 25334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:29.003424 25334 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [term 1 LEADER]: Becoming Leader. State: Replica: dbd59ae30e2e4bb8a24b9fd1ad96eba4, State: Running, Role: LEADER
I20260619 18:46:29.004011 25334 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } }
I20260619 18:46:29.004858 25324 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:29.006999 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } } }
I20260619 18:46:29.007462 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:29.010176 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd59ae30e2e4bb8a24b9fd1ad96eba4. Latest consensus state: current_term: 1 leader_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd59ae30e2e4bb8a24b9fd1ad96eba4" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 35161 } } }
I20260619 18:46:29.010661 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:29.011149 25343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:29.016932 25343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:29.027143 25343 catalog_manager.cc:1347] Generated new cluster ID: c814d57769554bd9b9e6ba7d49bf01bc
I20260619 18:46:29.027287 25343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:29.044921 25343 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:29.046993 25343 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:29.070322 25343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd59ae30e2e4bb8a24b9fd1ad96eba4: Generated new TSK 0
I20260619 18:46:29.071575 25343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:29.090277 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25257
W20260619 18:46:29.092108 25319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58963
W20260619 18:46:29.093297 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25257: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:29.093483 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25257
W20260619 18:46:29.111771 25242 connection.cc:535] client connection to 127.3.85.190:35161 recv error: Network error: failed to read from TLS socket (remote: 127.3.85.190:35161): Connection reset by peer (error 104)
2026-06-19T18:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:29 dist-test-slave-09fp krb5kdc[25355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:29 dist-test-slave-09fp krb5kdc[25355](info): set up 2 sockets
Jun 19 18:46:29 dist-test-slave-09fp krb5kdc[25355](info): commencing operation
krb5kdc: starting...
W20260619 18:46:30.235131 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:30 dist-test-slave-09fp krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:30.510643 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37703
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:37703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:30.870236 25368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:30.870720 25368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:30.870923 25368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:30.878856 25368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:30.879019 25368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:30.879125 25368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:30.879238 25368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:30.879334 25368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:30.889890 25368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:37703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:37703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:30.891364 25368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:30.892724 25368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:30.905468 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.905673 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:30.905652 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:30.906800 25368 server_base.cc:1034] running on GCE node
I20260619 18:46:30.908020 25368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:30.909873 25368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:30.911195 25368 hybrid_clock.cc:648] HybridClock initialized: now 1781894790911111 us; error 54 us; skew 500 ppm
I20260619 18:46:30.911644 25368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:30 dist-test-slave-09fp krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894790, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:30.915244 25368 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:30.917066 25368 webserver.cc:460] Webserver started at http://127.3.85.190:41759/ using document root <none> and password file <none>
I20260619 18:46:30.917924 25368 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:30.918095 25368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:30.918432 25368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:30.921777 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "2f036282f2ed4f80aaa43ca59077045b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-09fp"
server_key: "f6140790b49c233a36ea243c16fbe777"
server_key_iv: "2cfb5d24736210f4ea24988ba2f4ceba"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.922688 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "2f036282f2ed4f80aaa43ca59077045b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-09fp"
server_key: "f6140790b49c233a36ea243c16fbe777"
server_key_iv: "2cfb5d24736210f4ea24988ba2f4ceba"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.928553 25368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:30.932742 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:30.934391 25368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:46:30.934695 25368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "2f036282f2ed4f80aaa43ca59077045b"
format_stamp: "Formatted at 2026-06-19 18:46:30 on dist-test-slave-09fp"
server_key: "f6140790b49c233a36ea243c16fbe777"
server_key_iv: "2cfb5d24736210f4ea24988ba2f4ceba"
server_key_version: "encryptionkey@0"
I20260619 18:46:30.934912 25368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:30.941464 25368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:30.946168 25368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:30.946599 25368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:30.967415 25368 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:37703
I20260619 18:46:30.967478 25434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:37703 every 8 connection(s)
I20260619 18:46:30.969038 25368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:30.973322 25435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:30.979061 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25368
I20260619 18:46:30.979249 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:30.979808 3414 external_mini_cluster.cc:1428] Setting key dc3e2dba9eb609101cc00e163cd1cd5d
I20260619 18:46:30.986814 25435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b: Bootstrap starting.
I20260619 18:46:30.989049 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.981967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56251 (local address 127.3.85.190:37703)
0619 18:46:30.982889 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:30.982902 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:30.982947 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:46:30.985205 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:30.985237 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:30.985668 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:30.986154 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:30.987348 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:30.987356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:30.987369 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:30.987393 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:30.987456 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:30.987476 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:30.987736 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:30.987878 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 18:46:30.988073 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":431,"threads_started":1}
I20260619 18:46:30.990734 25435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:30.992061 25435 log.cc:824] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:30.993460 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:30.995957 25435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b: No bootstrap required, opened a new log
I20260619 18:46:31.002614 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:30.999828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51297 (local address 127.3.85.190:37703)
0619 18:46:31.000055 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:31.000061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:31.000374 (+ 313us) server_negotiation.cc:407] Connection header received
0619 18:46:31.000592 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:31.000596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:31.000655 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:31.000762 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:31.001827 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:31.001832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:31.001835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:31.001860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:31.001903 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:31.001919 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:31.002019 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:31.002152 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 18:46:31.002206 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:46:31.007084 25435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } }
I20260619 18:46:31.007535 25435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:31.007639 25435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f036282f2ed4f80aaa43ca59077045b, State: Initialized, Role: FOLLOWER
I20260619 18:46:31.008265 25435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } }
I20260619 18:46:31.008519 25435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:31.008684 25435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:31.008872 25435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:31.011313 25435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } }
I20260619 18:46:31.011971 25435 leader_election.cc:302] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f036282f2ed4f80aaa43ca59077045b; no voters:
I20260619 18:46:31.012642 25435 leader_election.cc:288] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:31.012817 25445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:31.013506 25445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [term 1 LEADER]: Becoming Leader. State: Replica: 2f036282f2ed4f80aaa43ca59077045b, State: Running, Role: LEADER
I20260619 18:46:31.014048 25445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } }
I20260619 18:46:31.014880 25435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:31.017365 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f036282f2ed4f80aaa43ca59077045b. Latest consensus state: current_term: 1 leader_uuid: "2f036282f2ed4f80aaa43ca59077045b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } } }
I20260619 18:46:31.017807 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:31.018217 25446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f036282f2ed4f80aaa43ca59077045b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f036282f2ed4f80aaa43ca59077045b" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 37703 } } }
I20260619 18:46:31.018652 25446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:31.031477 25454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:31.036466 25454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:31.046319 25454 catalog_manager.cc:1347] Generated new cluster ID: 96fbe9f3f18e46df8a3bacf5a9323d60
I20260619 18:46:31.046518 25454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:31.064992 25454 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:31.065527 25454 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:31.076908 25454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f036282f2ed4f80aaa43ca59077045b: Generated new TSK 0
I20260619 18:46:31.077929 25454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:31.097086 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25368
W20260619 18:46:31.098289 25430 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56251
W20260619 18:46:31.099259 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:31.099434 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25368
2026-06-19T18:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:31 dist-test-slave-09fp krb5kdc[25466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:31 dist-test-slave-09fp krb5kdc[25466](info): set up 2 sockets
Jun 19 18:46:31 dist-test-slave-09fp krb5kdc[25466](info): commencing operation
krb5kdc: starting...
W20260619 18:46:32.231101 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:32 dist-test-slave-09fp krb5kdc[25466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:32.526293 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39763
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:32.902400 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:32.902876 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:32.903048 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:32.911113 25479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:32.911305 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:32.911425 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:32.911538 25479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:32.911620 25479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:32.922276 25479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:32.923806 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:32.925294 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:32.938469 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:32.938625 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:32.938601 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:32.940110 25479 server_base.cc:1034] running on GCE node
I20260619 18:46:32.940987 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:32.942739 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:32.944016 25479 hybrid_clock.cc:648] HybridClock initialized: now 1781894792943934 us; error 67 us; skew 500 ppm
I20260619 18:46:32.944501 25479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:32 dist-test-slave-09fp krb5kdc[25466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:32.948318 25479 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:32.950205 25479 webserver.cc:460] Webserver started at http://127.3.85.190:43581/ using document root <none> and password file <none>
I20260619 18:46:32.951133 25479 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:32.951458 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:32.951846 25479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:32.955435 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "1d340f8472004b13bd4adbf41157301a"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-09fp"
server_key: "69982efac57d1f093069fc5027d84499"
server_key_iv: "722ded0c4df5018220f4e597bb01c6cc"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.956411 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "1d340f8472004b13bd4adbf41157301a"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-09fp"
server_key: "69982efac57d1f093069fc5027d84499"
server_key_iv: "722ded0c4df5018220f4e597bb01c6cc"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.962638 25479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260619 18:46:32.966952 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:32.968532 25479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:32.968776 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "1d340f8472004b13bd4adbf41157301a"
format_stamp: "Formatted at 2026-06-19 18:46:32 on dist-test-slave-09fp"
server_key: "69982efac57d1f093069fc5027d84499"
server_key_iv: "722ded0c4df5018220f4e597bb01c6cc"
server_key_version: "encryptionkey@0"
I20260619 18:46:32.969043 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:32.982957 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:32.988170 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:32.988549 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:33.010739 25479 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39763
I20260619 18:46:33.010776 25545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39763 every 8 connection(s)
I20260619 18:46:33.012530 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:33.016208 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25479
I20260619 18:46:33.016451 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:33.016901 3414 external_mini_cluster.cc:1428] Setting key 43b204d0ef5735231a43d67a0df26eb3
I20260619 18:46:33.016984 25546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:33.033346 25546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a: Bootstrap starting.
Jun 19 18:46:33 dist-test-slave-09fp krb5kdc[25466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:33.037791 25546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:33.039364 25546 log.cc:824] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:33.043519 25546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a: No bootstrap required, opened a new log
I20260619 18:46:33.045157 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:33.018820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56961 (local address 127.3.85.190:39763)
0619 18:46:33.019679 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:33.019693 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:33.019745 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:46:33.022047 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:33.022078 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:33.022674 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:33.023072 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:33.025082 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:33.026612 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:33.028492 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:33.029325 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:33.029484 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:33.035291 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:33.035304 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:33.035326 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:33.035403 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:33.039786 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:33.041016 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:33.041027 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:33.041037 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:33.041301 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:33.041735 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:33.041741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:33.041744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:33.042539 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:33.042862 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:33.043241 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 18:46:33.043742 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":416,"threads_started":1}
I20260619 18:46:33.049335 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:33.054124 25546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } }
I20260619 18:46:33.054709 25546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:33.054863 25546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d340f8472004b13bd4adbf41157301a, State: Initialized, Role: FOLLOWER
I20260619 18:46:33.055652 25546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } }
I20260619 18:46:33.055974 25546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:33.056134 25546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:33.056335 25546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:33.059322 25546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } }
I20260619 18:46:33.059937 25546 leader_election.cc:302] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d340f8472004b13bd4adbf41157301a; no voters:
I20260619 18:46:33.060633 25546 leader_election.cc:288] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:33.060793 25556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:33.062685 25556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [term 1 LEADER]: Becoming Leader. State: Replica: 1d340f8472004b13bd4adbf41157301a, State: Running, Role: LEADER
I20260619 18:46:33.063164 25546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:33.063390 25556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } }
I20260619 18:46:33.066407 25557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d340f8472004b13bd4adbf41157301a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } } }
I20260619 18:46:33.066917 25557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:33.069240 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d340f8472004b13bd4adbf41157301a. Latest consensus state: current_term: 1 leader_uuid: "1d340f8472004b13bd4adbf41157301a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d340f8472004b13bd4adbf41157301a" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39763 } } }
I20260619 18:46:33.069679 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:33.070947 25562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:33.074033 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:33.054725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45481 (local address 127.3.85.190:39763)
0619 18:46:33.054946 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:33.054953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:33.055530 (+ 577us) server_negotiation.cc:407] Connection header received
0619 18:46:33.055658 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:33.055665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:33.055761 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:33.055889 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:33.057574 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:33.058755 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:33.060107 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:33.060874 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:33.061714 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:33.064437 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:33.064444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:33.064449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:33.064494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:33.069239 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:33.070353 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:33.070357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:33.070361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:33.070550 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:33.071009 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:33.071017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:33.071024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:33.071573 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:33.072951 (+ 1378us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:33.073128 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 18:46:33.073452 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:46:33.077819 25562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:33.087059 25562 catalog_manager.cc:1347] Generated new cluster ID: c8f25fca365a488bb56e322383713e08
I20260619 18:46:33.087203 25562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:33.105741 25562 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:33.107312 25562 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:33.117383 25562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d340f8472004b13bd4adbf41157301a: Generated new TSK 0
I20260619 18:46:33.118669 25562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:33.124912 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25479
I20260619 18:46:33.126483 25541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:33.218181 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25479
2026-06-19T18:46:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:33 dist-test-slave-09fp krb5kdc[25578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:33 dist-test-slave-09fp krb5kdc[25578](info): set up 2 sockets
Jun 19 18:46:33 dist-test-slave-09fp krb5kdc[25578](info): commencing operation
krb5kdc: starting...
W20260619 18:46:34.335002 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:34 dist-test-slave-09fp krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:34.628182 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44251
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:44251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:34.988225 25591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:34.988565 25591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:34.988713 25591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:34.996522 25591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:34.996656 25591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:34.996749 25591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:34.996866 25591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:34.996958 25591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:35.008458 25591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:44251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:35.009956 25591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:35.012024 25591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:35.025393 25596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:35.025506 25597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:35.025655 25599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:35.028404 25591 server_base.cc:1034] running on GCE node
I20260619 18:46:35.029403 25591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:35.031291 25591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:35.032588 25591 hybrid_clock.cc:648] HybridClock initialized: now 1781894795032554 us; error 86 us; skew 500 ppm
I20260619 18:46:35.033000 25591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:35 dist-test-slave-09fp krb5kdc[25578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:35.040220 25591 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:35.042188 25591 webserver.cc:460] Webserver started at http://127.3.85.190:43667/ using document root <none> and password file <none>
I20260619 18:46:35.043123 25591 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:35.043300 25591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:35.043644 25591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:35.047096 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "4d82f874174a4ca1b716f2a4e29dcfae"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-09fp"
server_key: "51274a81664cf6779f823dfdb9e0c2f3"
server_key_iv: "b7c1e8946170b3d2dd40b0165ee91101"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.048074 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "4d82f874174a4ca1b716f2a4e29dcfae"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-09fp"
server_key: "51274a81664cf6779f823dfdb9e0c2f3"
server_key_iv: "b7c1e8946170b3d2dd40b0165ee91101"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.054085 25591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 18:46:35.058240 25606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:35.059821 25591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:35.060060 25591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "4d82f874174a4ca1b716f2a4e29dcfae"
format_stamp: "Formatted at 2026-06-19 18:46:35 on dist-test-slave-09fp"
server_key: "51274a81664cf6779f823dfdb9e0c2f3"
server_key_iv: "b7c1e8946170b3d2dd40b0165ee91101"
server_key_version: "encryptionkey@0"
I20260619 18:46:35.060307 25591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:35.096570 25591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:35.101768 25591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:35.102121 25591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:35.122785 25591 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:44251
I20260619 18:46:35.122802 25657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:44251 every 8 connection(s)
I20260619 18:46:35.124362 25591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:35.128825 25658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:35.128906 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25591
I20260619 18:46:35.129070 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:35.129607 3414 external_mini_cluster.cc:1428] Setting key 7b0d60ab4c66dc5db5a817d793cae8d9
I20260619 18:46:35.143221 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae: Bootstrap starting.
I20260619 18:46:35.144198 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:35.131634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60081 (local address 127.3.85.190:44251)
0619 18:46:35.132697 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:35.132715 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:35.132777 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:46:35.134630 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:35.134661 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:35.135182 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:35.135645 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:35.136281 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:35.137601 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:35.139112 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:35.139891 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:35.140142 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:35.141509 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:35.141518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:35.141537 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:35.142142 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:35.142237 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:35.142267 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:35.142426 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:35.142707 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:46:35.143033 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":517,"threads_started":1}
I20260619 18:46:35.147143 25658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:35.148926 25658 log.cc:824] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:35.149288 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:35.152469 25658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae: No bootstrap required, opened a new log
I20260619 18:46:35.161705 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:35.154424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34797 (local address 127.3.85.190:44251)
0619 18:46:35.154658 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:35.154666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:35.155071 (+ 405us) server_negotiation.cc:407] Connection header received
0619 18:46:35.155251 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:35.155257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:35.155379 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:35.155510 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:35.156026 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:35.157156 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:35.158662 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:35.159381 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:35.159528 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:35.160810 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:35.160818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:35.160823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:35.160849 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:35.160907 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:35.160935 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:35.161039 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:35.161221 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 18:46:35.161330 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:46:35.162986 25658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } }
I20260619 18:46:35.163520 25658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:35.163641 25658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d82f874174a4ca1b716f2a4e29dcfae, State: Initialized, Role: FOLLOWER
I20260619 18:46:35.164251 25658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } }
I20260619 18:46:35.164538 25658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:35.164726 25658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:35.164932 25658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:35.167454 25658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } }
I20260619 18:46:35.168021 25658 leader_election.cc:302] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d82f874174a4ca1b716f2a4e29dcfae; no voters:
I20260619 18:46:35.168665 25658 leader_election.cc:288] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:35.168948 25668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:35.170035 25668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [term 1 LEADER]: Becoming Leader. State: Replica: 4d82f874174a4ca1b716f2a4e29dcfae, State: Running, Role: LEADER
I20260619 18:46:35.170706 25668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } }
I20260619 18:46:35.171097 25658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:35.173724 25669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } } }
I20260619 18:46:35.173820 25670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d82f874174a4ca1b716f2a4e29dcfae. Latest consensus state: current_term: 1 leader_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d82f874174a4ca1b716f2a4e29dcfae" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44251 } } }
I20260619 18:46:35.174146 25669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:35.174212 25670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:35.175155 25676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:35.182560 25676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:35.195967 25676 catalog_manager.cc:1347] Generated new cluster ID: a33468cde2894c4ab3df1cf39d11313c
I20260619 18:46:35.196106 25676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:35.209074 25676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:35.211377 25676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:35.223093 25676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d82f874174a4ca1b716f2a4e29dcfae: Generated new TSK 0
I20260619 18:46:35.224562 25676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:35.232214 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25591
W20260619 18:46:35.234642 25653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60081
W20260619 18:46:35.235582 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:35.235786 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25591
2026-06-19T18:46:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:35 dist-test-slave-09fp krb5kdc[25689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:35 dist-test-slave-09fp krb5kdc[25689](info): set up 2 sockets
Jun 19 18:46:35 dist-test-slave-09fp krb5kdc[25689](info): commencing operation
krb5kdc: starting...
W20260619 18:46:36.371315 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:36 dist-test-slave-09fp krb5kdc[25689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:36.651170 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44921
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:34541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:37.005471 25702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:37.005807 25702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:37.005903 25702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:37.013543 25702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:37.013689 25702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:37.013737 25702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:37.013800 25702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:37.013846 25702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:37.025722 25702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:34541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:37.027195 25702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:37.028513 25702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:37.041857 25708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:37.041790 25710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:37.042389 25707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:37.042985 25702 server_base.cc:1034] running on GCE node
I20260619 18:46:37.043958 25702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:37.045678 25702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:37.046963 25702 hybrid_clock.cc:648] HybridClock initialized: now 1781894797046924 us; error 84 us; skew 500 ppm
I20260619 18:46:37.047461 25702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:37 dist-test-slave-09fp krb5kdc[25689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:37.051123 25702 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:37.053143 25702 webserver.cc:460] Webserver started at http://127.3.85.190:46083/ using document root <none> and password file <none>
I20260619 18:46:37.053967 25702 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:37.054122 25702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:37.054538 25702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:37.057814 25702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3"
format_stamp: "Formatted at 2026-06-19 18:46:37 on dist-test-slave-09fp"
server_key: "c4f7998817d27031ff3a1daba7075b69"
server_key_iv: "863bb7cfeb108a0f7a3dad5bd050bb3d"
server_key_version: "encryptionkey@0"
I20260619 18:46:37.058768 25702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3"
format_stamp: "Formatted at 2026-06-19 18:46:37 on dist-test-slave-09fp"
server_key: "c4f7998817d27031ff3a1daba7075b69"
server_key_iv: "863bb7cfeb108a0f7a3dad5bd050bb3d"
server_key_version: "encryptionkey@0"
I20260619 18:46:37.065136 25702 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:46:37.069612 25717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:37.071429 25702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 18:46:37.071667 25702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3"
format_stamp: "Formatted at 2026-06-19 18:46:37 on dist-test-slave-09fp"
server_key: "c4f7998817d27031ff3a1daba7075b69"
server_key_iv: "863bb7cfeb108a0f7a3dad5bd050bb3d"
server_key_version: "encryptionkey@0"
I20260619 18:46:37.071892 25702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:37.078944 25702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:37.083881 25702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:37.084235 25702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:37.104063 25702 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:44921
I20260619 18:46:37.104214 25768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:44921 every 8 connection(s)
I20260619 18:46:37.105621 25702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:37.109171 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25702
I20260619 18:46:37.109486 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:37.110163 3414 external_mini_cluster.cc:1428] Setting key eeddb3a23df85a1bd51037818d2d7143
I20260619 18:46:37.110484 25769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:37.121206 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:37.113195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34281 (local address 127.3.85.190:44921)
0619 18:46:37.114346 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:37.114359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:37.114405 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:46:37.116870 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:37.116903 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:37.117349 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:37.117738 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:37.119107 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:37.119115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:37.119131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:37.119151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:37.119224 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:37.119253 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:37.119463 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:37.119586 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:46:37.119897 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":461,"threads_started":1}
I20260619 18:46:37.125332 25769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3: Bootstrap starting.
I20260619 18:46:37.125921 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:37.129211 25769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:37.130589 25769 log.cc:824] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:37.133836 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:37.130991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45059 (local address 127.3.85.190:44921)
0619 18:46:37.131255 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:37.131264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:37.131623 (+ 359us) server_negotiation.cc:407] Connection header received
0619 18:46:37.132001 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:37.132011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:37.132064 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:37.132187 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:37.133174 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:37.133180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:37.133184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:46:37.133203 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:37.133251 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:46:37.133267 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:37.133400 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:37.133486 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 18:46:37.133551 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:46:37.134544 25769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3: No bootstrap required, opened a new log
I20260619 18:46:37.142813 25769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } }
I20260619 18:46:37.143277 25769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:37.143393 25769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380c1fbf195b4ccaa61e0447ee8ad0a3, State: Initialized, Role: FOLLOWER
I20260619 18:46:37.144116 25769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } }
I20260619 18:46:37.144398 25769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:37.144554 25769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:37.144749 25769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:37.147557 25769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } }
I20260619 18:46:37.148135 25769 leader_election.cc:302] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380c1fbf195b4ccaa61e0447ee8ad0a3; no voters:
I20260619 18:46:37.148808 25769 leader_election.cc:288] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:37.149009 25779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:37.149806 25779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 380c1fbf195b4ccaa61e0447ee8ad0a3, State: Running, Role: LEADER
I20260619 18:46:37.150898 25779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } }
I20260619 18:46:37.151151 25769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:37.154779 25780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } } }
I20260619 18:46:37.154732 25781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380c1fbf195b4ccaa61e0447ee8ad0a3. Latest consensus state: current_term: 1 leader_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380c1fbf195b4ccaa61e0447ee8ad0a3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 44921 } } }
I20260619 18:46:37.155316 25781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:37.155318 25780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:37.156420 25788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:37.162356 25788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:37.172665 25788 catalog_manager.cc:1347] Generated new cluster ID: eadae685ce0742ae96755a63d5db58f1
I20260619 18:46:37.172816 25788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:37.192320 25788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:37.192834 25788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:37.201496 25788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380c1fbf195b4ccaa61e0447ee8ad0a3: Generated new TSK 0
I20260619 18:46:37.202765 25788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:37.219066 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25702
W20260619 18:46:37.220798 25764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34281
W20260619 18:46:37.221728 3414 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:46:37.221917 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25702
2026-06-19T18:46:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1989 ms)
[----------] 18 tests from AuthTokenIssuingTest (36568 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:37 dist-test-slave-09fp krb5kdc[25800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:37 dist-test-slave-09fp krb5kdc[25800](info): set up 2 sockets
Jun 19 18:46:37 dist-test-slave-09fp krb5kdc[25800](info): commencing operation
krb5kdc: starting...
W20260619 18:46:38.423223 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:38 dist-test-slave-09fp krb5kdc[25800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:38.706326 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34599
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:39.074173 25813 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:39.074690 25813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:39.074837 25813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:39.074990 25813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:39.082973 25813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:39.083154 25813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:39.083272 25813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:39.083374 25813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:39.083496 25813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:39.094324 25813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:34599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:39.095890 25813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:39.097522 25813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:39.111115 25818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.111231 25819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.111446 25821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:39.112471 25813 server_base.cc:1034] running on GCE node
I20260619 18:46:39.113551 25813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:39.115370 25813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:39.116664 25813 hybrid_clock.cc:648] HybridClock initialized: now 1781894799116596 us; error 45 us; skew 500 ppm
I20260619 18:46:39.117218 25813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:39.121237 25813 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:39.123215 25813 webserver.cc:460] Webserver started at http://127.3.85.190:46691/ using document root <none> and password file <none>
I20260619 18:46:39.124042 25813 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:39.124166 25813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:39.124536 25813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:39.128031 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "bb09ee5859d648e9816844e4a6a218f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "a0dce88cebd5f9813e2dbff2a4bd0928"
server_key_iv: "c3d845b1d5c727424b6e36df960369bd"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.128975 25813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "bb09ee5859d648e9816844e4a6a218f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "a0dce88cebd5f9813e2dbff2a4bd0928"
server_key_iv: "c3d845b1d5c727424b6e36df960369bd"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.134891 25813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 18:46:39.138980 25828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:39.140671 25813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:46:39.140905 25813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "bb09ee5859d648e9816844e4a6a218f3"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "a0dce88cebd5f9813e2dbff2a4bd0928"
server_key_iv: "c3d845b1d5c727424b6e36df960369bd"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.141147 25813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:39.157549 25813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:39.162432 25813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:39.162784 25813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:39.183400 25813 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:34599
I20260619 18:46:39.183436 25879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:34599 every 8 connection(s)
I20260619 18:46:39.185070 25813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:39.188740 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25813
I20260619 18:46:39.189040 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:39.189503 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:39.189821 3414 external_mini_cluster.cc:1428] Setting key 8af6c2a6c1ffd3ab140795d88e972302
I20260619 18:46:39.197703 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.85.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.189@KRBTEST.COM" created.
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:39.228674 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.201797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34599 (local address 127.0.0.1:42974)
0619 18:46:39.202910 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.202958 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:39.205308 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:39.206345 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:39.206352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:39.207157 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:39.207511 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.207531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.209180 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.209189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.210400 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.210433 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.212131 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.212141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.212383 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.213341 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:39.213379 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:39.219879 (+ 6500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:39.223871 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.223892 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.223910 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.224616 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.225194 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.225202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.225209 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.225451 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.226398 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:39.226410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:39.226735 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 18:46:39.226973 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 18:46:39.227456 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":480,"threads_started":1}
I20260619 18:46:39.228722 25884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.203514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42974 (local address 127.3.85.190:34599)
0619 18:46:39.204320 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.204334 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.204410 (+ 76us) server_negotiation.cc:407] Connection header received
0619 18:46:39.205804 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.205811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.206088 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.206320 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.207767 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.208989 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.210660 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.211738 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.212101 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.220309 (+ 8208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.220319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.220335 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:39.220377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.223657 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.224836 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.224844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.224847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.225036 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.225596 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.225600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.225603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.226209 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.226408 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.227207 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.227707 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":580,"thread_start_us":402,"threads_started":1}
I20260619 18:46:39.232693 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:39.234226 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.233609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53768)
0619 18:46:39.233827 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.234074 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 18:46:39.234697 25831 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
I20260619 18:46:39.236972 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.236527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53780)
0619 18:46:39.236711 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.236810 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:46:39.237236 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:33471: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
I20260619 18:46:39.255303 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:33471
--webserver_interface=127.3.85.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:39.286645 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } attempt: 1
I20260619 18:46:39.287808 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.287337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53784)
0619 18:46:39.287555 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.287653 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 18:46:39.289251 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.288711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53788)
0619 18:46:39.288975 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.289068 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 18:46:39.289533 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:33471: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
I20260619 18:46:39.357838 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } attempt: 2
I20260619 18:46:39.359262 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.358768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53802)
0619 18:46:39.359020 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.359112 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:46:39.360496 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.360094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53806)
0619 18:46:39.360268 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.360356 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:46:39.360816 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:33471: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
I20260619 18:46:39.452095 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } attempt: 3
I20260619 18:46:39.453406 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.452952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53816)
0619 18:46:39.453166 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.453267 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:46:39.454746 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.454307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53818)
0619 18:46:39.454506 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.454584 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:46:39.455017 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:33471: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
I20260619 18:46:39.598343 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } attempt: 4
I20260619 18:46:39.599776 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.599194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53824)
0619 18:46:39.599545 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.599638 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260619 18:46:39.601202 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.600793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53826)
0619 18:46:39.600948 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.601059 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 18:46:39.601482 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:33471: Network error: Client connection negotiation failed: client connection to 127.3.85.189:33471: connect: Connection refused (error 111)
W20260619 18:46:39.623529 25887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:39.623945 25887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:39.624111 25887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:39.624219 25887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:39.632151 25887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:39.632268 25887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:39.632385 25887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:39.632475 25887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:39.632602 25887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:39.643474 25887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.85.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:39.645115 25887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:39.646767 25887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:39.660144 25893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.660261 25895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:39.660223 25892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:39.660990 25887 server_base.cc:1034] running on GCE node
I20260619 18:46:39.661855 25887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:39.663583 25887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:39.664872 25887 hybrid_clock.cc:648] HybridClock initialized: now 1781894799664807 us; error 41 us; skew 500 ppm
I20260619 18:46:39.665247 25887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:39.669122 25887 init.cc:377] Logged in from keytab as kudu/127.3.85.189@KRBTEST.COM (short username kudu)
I20260619 18:46:39.671615 25887 webserver.cc:460] Webserver started at http://127.3.85.189:46327/ using document root <none> and password file <none>
I20260619 18:46:39.672421 25887 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:39.672544 25887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:39.672856 25887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:39.676406 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data/instance:
uuid: "edc0d23ea86d4ebcb60a017cba91830a"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "b34201aa7915c3398856ef75ea5d4431"
server_key_iv: "1dae04f6cd064199ab69839c75ae4d31"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.677294 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal/instance:
uuid: "edc0d23ea86d4ebcb60a017cba91830a"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "b34201aa7915c3398856ef75ea5d4431"
server_key_iv: "1dae04f6cd064199ab69839c75ae4d31"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.683501 25887 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:46:39.687356 25902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:39.689019 25887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:39.689283 25887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "edc0d23ea86d4ebcb60a017cba91830a"
format_stamp: "Formatted at 2026-06-19 18:46:39 on dist-test-slave-09fp"
server_key: "b34201aa7915c3398856ef75ea5d4431"
server_key_iv: "1dae04f6cd064199ab69839c75ae4d31"
server_key_version: "encryptionkey@0"
I20260619 18:46:39.689520 25887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:39.713800 25887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:39.718969 25887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:39.719269 25887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:39.739302 25887 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.189:33471
I20260619 18:46:39.739346 25953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.189:33471 every 8 connection(s)
I20260619 18:46:39.740839 25887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
I20260619 18:46:39.743726 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25887
I20260619 18:46:39.743945 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-1/wal/instance
I20260619 18:46:39.744592 3414 external_mini_cluster.cc:1428] Setting key 99682b80533fe913a27cc55fc0776e1b
I20260619 18:46:39.746049 25954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:39.757714 25954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.85.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.188@KRBTEST.COM" created.
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:39.785167 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.760934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.3.85.190:34599)
0619 18:46:39.761554 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.761559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.762601 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 18:46:39.764672 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.764679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.764780 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.764911 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.766753 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.767546 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.769353 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.769800 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.770019 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.777307 (+ 7288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.777314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.777318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:39.777371 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.780697 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.781981 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.781987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.781990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.782148 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.782764 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.782769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.782771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.783108 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.783237 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.784294 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.784602 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":354,"threads_started":1}
I20260619 18:46:39.785771 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.761479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34599 (local address 127.0.0.1:42978)
0619 18:46:39.762356 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.762397 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:39.764353 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:39.765201 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:39.765211 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:39.765780 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:39.766411 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.766431 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.767748 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.767753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.769174 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.769189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.770147 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.770156 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.770414 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.771334 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:39.771365 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:39.776893 (+ 5528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:39.780950 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.780965 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.780991 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.781776 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.782297 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.782305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.782311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.782634 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.783257 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:39.783270 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:39.783837 (+ 567us) client_negotiation.cc:769] Sending connection context
0619 18:46:39.784090 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 18:46:39.784528 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":375,"threads_started":1}
I20260619 18:46:39.787715 25954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:39.805409 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.788594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53836)
0619 18:46:39.788821 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.788844 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:39.788956 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:39.790243 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:39.790248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:39.790794 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:39.790939 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.790948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.792484 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.792493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.793487 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.793502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.794544 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.794549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.794724 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.795504 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:39.795525 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:39.798372 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:39.802202 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.802211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.802215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.802849 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.803494 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.803498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.803500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.803638 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.804480 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:39.804485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:39.804648 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 18:46:39.804780 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:46:39.805105 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:46:39.805811 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.789002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.3.85.189:33471)
0619 18:46:39.789753 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.789759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.789788 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:46:39.789842 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.789848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.790064 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.790256 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.791127 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.792319 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.793647 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.794357 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.794540 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.798738 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.798754 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.798770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:39.798812 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.801993 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.803062 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.803073 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.803076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.803283 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.803781 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.803785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.803787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.804298 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.804498 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.805106 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.805406 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":390,"threads_started":1}
I20260619 18:46:39.808857 25954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
I20260619 18:46:39.810392 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.809704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:59982)
0619 18:46:39.809961 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.810169 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 18:46:39.810434 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:45725
--webserver_interface=127.3.85.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:39.810770 25906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:39.813158 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.812500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:59998)
0619 18:46:39.812843 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.812975 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 18:46:39.813472 25954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:39.862929 25954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 1
I20260619 18:46:39.864373 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.863779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60014)
0619 18:46:39.864091 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.864217 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 18:46:39.865571 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.865202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60016)
0619 18:46:39.865352 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.865454 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:46:39.865907 25954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:39.900841 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } attempt: 5
Jun 19 18:46:39 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:39.916242 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.901785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.3.85.189:33471)
0619 18:46:39.901997 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:39.902002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:39.902026 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:46:39.902131 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:39.902137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:39.902260 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:39.902377 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:39.903239 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.903975 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.905182 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.905762 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.905901 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.909809 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:39.909815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:39.909818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:39.909856 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:39.912778 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.913599 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.913608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.913615 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.913860 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:39.914483 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:39.914491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:39.914495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:39.914931 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:39.915100 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:39.915629 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 18:46:39.915866 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:46:39.916227 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.901644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53844)
0619 18:46:39.901852 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.901875 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:39.902008 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:39.902421 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:39.902429 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:39.902877 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:39.903061 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.903071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.904124 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.904130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.905013 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:39.905025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:39.905986 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:39.905993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:39.906268 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:39.907002 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:39.907026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:39.909545 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:39.912970 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.912976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.912981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.913424 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.914063 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:39.914069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:39.914073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:39.914298 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:39.915062 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:39.915067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:39.915310 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 18:46:39.915518 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 18:46:39.915859 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:46:39.917912 25880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:39.919401 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.918868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60030)
0619 18:46:39.919049 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.919186 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:46:39.920590 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.920207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60046)
0619 18:46:39.920371 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.920469 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 18:46:39.920836 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:39.934175 25954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 2
I20260619 18:46:39.935524 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.934957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60048)
0619 18:46:39.935215 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.935322 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 18:46:39.936728 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.936307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60064)
0619 18:46:39.936480 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.936583 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:46:39.937004 25954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:39.972105 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 1
I20260619 18:46:39.973333 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.972844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60070)
0619 18:46:39.973058 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.973158 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 18:46:39.974543 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:39.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60076)
0619 18:46:39.974281 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:39.974354 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:46:39.974759 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:40.028339 25954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 3
I20260619 18:46:40.029789 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.029265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60092)
0619 18:46:40.029526 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.029623 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 18:46:40.031232 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.030651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60096)
0619 18:46:40.030984 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.031110 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260619 18:46:40.031513 25954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:40.043027 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 2
I20260619 18:46:40.044487 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.043800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60110)
0619 18:46:40.044076 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.044224 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:46:40.045984 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.045340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60118)
0619 18:46:40.045656 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.045780 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 18:46:40.046280 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:40.152608 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 3
I20260619 18:46:40.154017 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.153481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60128)
0619 18:46:40.153741 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.153839 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 18:46:40.155396 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.154950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60144)
0619 18:46:40.155189 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.155273 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 18:46:40.155682 25880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
I20260619 18:46:40.174808 25954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 4
I20260619 18:46:40.176016 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.175437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60148)
0619 18:46:40.175714 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.175828 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 18:46:40.177160 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.176768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60152)
0619 18:46:40.176940 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.177026 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:46:40.177412 25954 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:45725: Network error: Client connection negotiation failed: client connection to 127.3.85.188:45725: connect: Connection refused (error 111)
W20260619 18:46:40.179122 25962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:40.179451 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:40.179524 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:40.179606 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:40.187530 25962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:40.187630 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:40.187726 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:40.187820 25962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:40.187927 25962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:40.199536 25962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:34599,127.3.85.189:33471,127.3.85.188:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.85.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:40.200950 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:40.202404 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:40.215828 25967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:40.215801 25968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:40.216181 25970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:40.217269 25962 server_base.cc:1034] running on GCE node
I20260619 18:46:40.218197 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:40.219820 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:40.221076 25962 hybrid_clock.cc:648] HybridClock initialized: now 1781894800220991 us; error 60 us; skew 500 ppm
I20260619 18:46:40.221494 25962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:40.229773 25962 init.cc:377] Logged in from keytab as kudu/127.3.85.188@KRBTEST.COM (short username kudu)
I20260619 18:46:40.233081 25962 webserver.cc:460] Webserver started at http://127.3.85.188:46175/ using document root <none> and password file <none>
I20260619 18:46:40.234339 25962 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:40.234555 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:40.235158 25962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:40.239840 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data/instance:
uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d"
format_stamp: "Formatted at 2026-06-19 18:46:40 on dist-test-slave-09fp"
server_key: "f6cf04d53b7b072dbba913e9e79c06f0"
server_key_iv: "e16b4726a2d594696c9af09aabdc5740"
server_key_version: "encryptionkey@0"
I20260619 18:46:40.240711 25962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal/instance:
uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d"
format_stamp: "Formatted at 2026-06-19 18:46:40 on dist-test-slave-09fp"
server_key: "f6cf04d53b7b072dbba913e9e79c06f0"
server_key_iv: "e16b4726a2d594696c9af09aabdc5740"
server_key_version: "encryptionkey@0"
I20260619 18:46:40.246649 25962 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:46:40.250327 25977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:40.251888 25962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:40.252096 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d"
format_stamp: "Formatted at 2026-06-19 18:46:40 on dist-test-slave-09fp"
server_key: "f6cf04d53b7b072dbba913e9e79c06f0"
server_key_iv: "e16b4726a2d594696c9af09aabdc5740"
server_key_version: "encryptionkey@0"
I20260619 18:46:40.252355 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:40.272745 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:40.277329 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:40.277632 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:40.298179 25962 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.188:45725
I20260619 18:46:40.298193 26028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.188:45725 every 8 connection(s)
I20260619 18:46:40.299690 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
I20260619 18:46:40.303752 26029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:40.309478 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 25962
I20260619 18:46:40.309667 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894429649633-3414-0/minicluster-data/master-2/wal/instance
I20260619 18:46:40.310189 3414 external_mini_cluster.cc:1428] Setting key dce52eff11512d07918339c3cdb62cda
I20260619 18:46:40.311658 26029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:46:40.332928 25880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 4
I20260619 18:46:40.335309 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.314794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.3.85.190:34599)
0619 18:46:40.316418 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.316424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.316446 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:46:40.316515 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.316520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.316634 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.316745 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.318714 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.319851 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.322142 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.322853 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.322993 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.327524 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.327530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.327534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.327573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.330676 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.331490 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.331503 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.331506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.331644 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.332104 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.332110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.332113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.332400 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.332576 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.334688 (+ 2112us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.334928 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1333,"threads_started":1}
I20260619 18:46:40.335472 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.312417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.3.85.188:45725)
0619 18:46:40.313332 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.313345 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.313393 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:46:40.315904 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.315932 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.316468 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.316714 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.318715 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.320516 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.322342 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.323303 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.323402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.327976 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.327986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.328001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.328050 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.331065 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.331878 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.331893 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.331896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.332030 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.332438 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.332445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.332448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.332938 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.333394 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.333574 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.334412 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":582,"server-negotiator.queue_time_us":684,"spinlock_wait_cycles":23808,"thread_start_us":423,"threads_started":1}
I20260619 18:46:40.335834 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.314764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:34599 (local address 127.0.0.1:42992)
0619 18:46:40.315793 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.315856 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:40.316264 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:40.316849 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:40.316856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:40.317801 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:40.318414 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.318472 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.320050 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.320054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.321938 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.321957 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.323092 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.323097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.323319 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.324203 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:40.324239 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:40.327272 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:40.330854 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.330867 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.330884 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.331347 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.331773 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.331778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.331782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.331988 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.332542 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:40.332550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:40.333006 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 18:46:40.333337 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 18:46:40.333925 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"spinlock_wait_cycles":38144,"thread_start_us":399,"threads_started":1}
I20260619 18:46:40.338806 26029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:40.358146 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.333567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:60182)
0619 18:46:40.333939 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.333960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:40.334059 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:40.338291 (+ 4232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:40.338300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:40.339012 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:40.339314 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.339332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.341398 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.341408 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.343477 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.343501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.345042 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.345052 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.345384 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.346264 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:40.346299 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:40.349901 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:40.354515 (+ 4614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.354522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.354526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.355047 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.355543 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.355549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.355553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.355793 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.356729 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:40.356736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:40.357031 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 18:46:40.357254 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 18:46:40.357645 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 18:46:40.358146 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.334137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.3.85.188:45725)
0619 18:46:40.337839 (+ 3702us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.337848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.337872 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:46:40.337946 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.337954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.338056 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.339547 (+ 1491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.339749 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.341192 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.343769 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.344792 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.345934 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.350167 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.350172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.350175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.350252 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.354305 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.355220 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.355226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.355232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.355385 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.356047 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.356054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.356058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.356562 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.357037 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.357380 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.357674 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3546,"thread_start_us":15151,"threads_started":1}
I20260619 18:46:40.361071 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.339722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:33471 (local address 127.0.0.1:53860)
0619 18:46:40.340623 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.340640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:40.340930 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:40.341472 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:40.341478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:40.342175 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:40.342487 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.342504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.344120 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.344126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.346480 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.346502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.347904 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.347909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.348148 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.349495 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:40.349524 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:40.352900 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:40.357156 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.357162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.357167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.357815 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.358811 (+ 996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.358817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.358823 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.359016 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.359883 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:40.359888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:40.360092 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 18:46:40.360287 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:46:40.360613 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"spinlock_wait_cycles":1498496}
I20260619 18:46:40.361099 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.340125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.3.85.189:33471)
0619 18:46:40.340300 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.340308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.340823 (+ 515us) server_negotiation.cc:407] Connection header received
0619 18:46:40.341076 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.341082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.341160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.341230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.342715 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.343905 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.346699 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.347472 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.347601 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.353190 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.353196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.353199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.353240 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.356972 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.357985 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.357990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.357994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.358171 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.359170 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.359176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.359180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.359610 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.359803 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.360423 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.360676 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:46:40.362159 26029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:40.363853 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.340076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.3.85.190:34599)
0619 18:46:40.340556 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.340562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.340577 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:46:40.340626 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.340731 (+ 105us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.340859 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.340959 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.343105 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.344283 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.346654 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.347482 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.347620 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.354401 (+ 6781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.354408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.354413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.354498 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.358996 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.360092 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.360098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.360101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.360275 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.360794 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.360799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.360803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.361626 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.361965 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.363219 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.363482 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260619 18:46:40.365190 25880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: Bootstrap starting.
I20260619 18:46:40.369568 25880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:40.371042 25880 log.cc:824] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894800, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:46:40.374691 25880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: No bootstrap required, opened a new log
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:40.379743 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.363185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:39866)
0619 18:46:40.364190 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.364207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:40.364349 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:40.364863 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:40.364868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:40.365395 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:40.365623 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.365637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.366904 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.366913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.367843 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.367855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.368876 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.368882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.369108 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.369868 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:40.369887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:40.372498 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:40.376387 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.376391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.376396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.376943 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.377356 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.377362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.377367 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.377503 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.378317 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:40.378323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:40.378586 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 18:46:40.378771 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 18:46:40.379171 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":28}
I20260619 18:46:40.379768 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.362992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.3.85.188:45725)
0619 18:46:40.363449 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.363455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.364284 (+ 829us) server_negotiation.cc:407] Connection header received
0619 18:46:40.364499 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.364504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.364584 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.364686 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.365788 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.366755 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.368016 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.368701 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.368847 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.372735 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.372742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.372746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.372783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.376208 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.377087 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.377092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.377095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.377225 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.377634 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.377640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.377645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.378160 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.378349 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.378874 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.379303 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":311}
I20260619 18:46:40.386067 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.365833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.3.85.189:33471)
0619 18:46:40.365996 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.366001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.366247 (+ 246us) server_negotiation.cc:407] Connection header received
0619 18:46:40.366378 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.366385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.366491 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.366605 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.368294 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.369318 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.371258 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.372081 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.372951 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.377410 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.377417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.377420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.377460 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.382892 (+ 5432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.383761 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.383768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.383772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.383966 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.384403 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.384411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.384414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.384737 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.385051 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.385379 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.385645 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:46:40.386083 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: Bootstrap starting.
I20260619 18:46:40.386121 25880 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.386643 25880 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb09ee5859d648e9816844e4a6a218f3, State: Initialized, Role: FOLLOWER
I20260619 18:46:40.387188 25880 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.388314 26037 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.388720 26037 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.389700 25880 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:40.391533 26029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:40.392158 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:40.393491 26029 log.cc:824] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:40.395771 25855 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb09ee5859d648e9816844e4a6a218f3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43008
I20260619 18:46:40.396323 25855 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:46:40.396576 25855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:40.397538 26029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: No bootstrap required, opened a new log
I20260619 18:46:40.401295 25855 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.403954 25855 leader_election.cc:288] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [CANDIDATE]: Term 1 election: Requested vote from peers edc0d23ea86d4ebcb60a017cba91830a (127.3.85.189:33471), e5adc4d636d74ec0b329d67dc0ac4d1d (127.3.85.188:45725)
I20260619 18:46:40.409044 26029 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.409739 26029 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5adc4d636d74ec0b329d67dc0ac4d1d, State: Initialized, Role: FOLLOWER
I20260619 18:46:40.410379 26029 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.412575 26049 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:46:40.413630 26052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.414211 26052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.415222 26029 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:46:40.415944 25834 tablet.cc:2369] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:40.417768 26004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb09ee5859d648e9816844e4a6a218f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d"
I20260619 18:46:40.418613 26004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:40.425859 26004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb09ee5859d648e9816844e4a6a218f3 in term 1.
I20260619 18:46:40.427168 25832 leader_election.cc:302] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb09ee5859d648e9816844e4a6a218f3, e5adc4d636d74ec0b329d67dc0ac4d1d; no voters:
I20260619 18:46:40.427953 26037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:40.423141 25813 master_runner.cc:186] Error getting master registration for 127.3.85.189:33471: OK, instance_id { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" instance_seqno: 1781894799733553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:46:40.441699 25813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:40.447927 26037 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [term 1 LEADER]: Becoming Leader. State: Replica: bb09ee5859d648e9816844e4a6a218f3, State: Running, Role: LEADER
I20260619 18:46:40.449855 26037 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.475304 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.428614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.3.85.189:33471)
0619 18:46:40.428853 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.428860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.446831 (+ 17971us) server_negotiation.cc:407] Connection header received
0619 18:46:40.448369 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.448374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.448460 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.448566 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.450994 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.452198 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.453941 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.454610 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.454757 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.460045 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.460056 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.460064 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.460138 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.465591 (+ 5453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.471533 (+ 5942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.471540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.471544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.471790 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.472376 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.472385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.472391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.472937 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.473339 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.474230 (+ 891us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.474665 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:46:40.476611 25954 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } attempt: 5
Jun 19 18:46:40 dist-test-slave-09fp krb5kdc[25800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894799, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:46:40.501466 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.436145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.3.85.188:45725)
0619 18:46:40.436479 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.436490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.447101 (+ 10611us) server_negotiation.cc:407] Connection header received
0619 18:46:40.447315 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.447326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.447459 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.447584 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.451776 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.453244 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.458209 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.459252 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.459446 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.469207 (+ 9761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.469220 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.469227 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.469298 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.490711 (+ 21413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.492527 (+ 1816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.492538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.492544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.493109 (+ 565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.494594 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.494599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.494602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.495771 (+ 1169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.499083 (+ 3312us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.499505 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.500348 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 18:46:40.514068 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.477336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:45725 (local address 127.0.0.1:39884)
0619 18:46:40.478524 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:40.478550 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:40.478697 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:40.479630 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:40.479639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:40.480409 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:40.480758 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.480780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.483324 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.483335 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.485440 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:40.485469 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.487470 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.487481 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:40.487850 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.491431 (+ 3581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:40.491480 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:40.495903 (+ 4423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:40.505347 (+ 9444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.505356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.505363 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.506247 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.509751 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:40.509761 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:40.509769 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:40.510061 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:40.512160 (+ 2099us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:40.512170 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:40.512497 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 18:46:40.512761 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 18:46:40.513314 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976}
I20260619 18:46:40.516417 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.424342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.3.85.190:34599)
0619 18:46:40.426538 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.426548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.445950 (+ 19402us) server_negotiation.cc:407] Connection header received
0619 18:46:40.446216 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.446227 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.446370 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.446633 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.453516 (+ 6883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.455016 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.463522 (+ 8506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.464600 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.464780 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.469849 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.469863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.469872 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.469947 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.488958 (+ 19011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.506579 (+ 17621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.506600 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.506607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.507445 (+ 838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.508229 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.508238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.508245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.508811 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.509170 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.515237 (+ 6067us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.515662 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1954}
I20260619 18:46:40.521083 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.478482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.3.85.188:45725)
0619 18:46:40.478771 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.478782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.478823 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:46:40.478913 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.478922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.479040 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.479200 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.481135 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.483014 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.485773 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:40.486898 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:40.487106 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:40.497607 (+ 10501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:40.497621 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:40.497630 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:40.497702 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:40.505058 (+ 7356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.509286 (+ 4228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.509295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.509301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.509532 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:40.511001 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:40.511011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:40.511016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:40.511575 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:40.511929 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:40.519682 (+ 7753us) server_negotiation.cc:299] Negotiation successful
0619 18:46:40.520318 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 18:46:40.525708 26063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb09ee5859d648e9816844e4a6a218f3. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.526327 26063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:40.528302 25954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: Bootstrap starting.
W20260619 18:46:40.530043 25983 tablet.cc:2369] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:40.537734 25954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:40.539268 25954 log.cc:824] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:40.544456 25954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: No bootstrap required, opened a new log
I20260619 18:46:40.544574 26070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 18:46:40.560098 26071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:40.560386 26071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:40.560231 25954 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.562688 25954 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc0d23ea86d4ebcb60a017cba91830a, State: Initialized, Role: FOLLOWER
I20260619 18:46:40.563408 25954 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } }
I20260619 18:46:40.564206 26070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:40.572564 26075 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.575923 26075 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.575834 25929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb09ee5859d648e9816844e4a6a218f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "edc0d23ea86d4ebcb60a017cba91830a"
I20260619 18:46:40.576419 25929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:40.580049 25929 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb09ee5859d648e9816844e4a6a218f3 in term 1.
I20260619 18:46:40.575748 25954 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:40.568666 25962 master_runner.cc:186] Error getting master registration for 127.3.85.189:33471: OK, instance_id { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" instance_seqno: 1781894799733553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:46:40.584904 25962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:40.597105 26004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [term 1 FOLLOWER]: Refusing update from remote peer bb09ee5859d648e9816844e4a6a218f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:40.600291 26063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:40.604589 25929 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [term 1 FOLLOWER]: Refusing update from remote peer bb09ee5859d648e9816844e4a6a218f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:40.610602 26063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:40.647490 26052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb09ee5859d648e9816844e4a6a218f3. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.648021 26052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.652176 26077 mvcc.cc:205] Tried to move back new op lower bound from 7298641103226327040 to 7298641102657363968. Current Snapshot: MvccSnapshot[applied={T|T < 7298641103226327040}]
I20260619 18:46:40.654780 26075 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb09ee5859d648e9816844e4a6a218f3. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.655205 26075 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.658538 26080 mvcc.cc:205] Tried to move back new op lower bound from 7298641103226327040 to 7298641102657363968. Current Snapshot: MvccSnapshot[applied={T|T < 7298641103226327040}]
I20260619 18:46:40.668817 26075 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.669126 26075 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.671890 26063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.672242 26063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:40.673213 26063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.673470 26063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:40.675215 26070 catalog_manager.cc:1347] Generated new cluster ID: 2af94d4d99ba4519aabda7a37b457344
I20260619 18:46:40.675370 26070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:40.678977 26052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb09ee5859d648e9816844e4a6a218f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb09ee5859d648e9816844e4a6a218f3" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 34599 } } peers { permanent_uuid: "edc0d23ea86d4ebcb60a017cba91830a" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 33471 } } peers { permanent_uuid: "e5adc4d636d74ec0b329d67dc0ac4d1d" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 45725 } } }
I20260619 18:46:40.679293 26052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:40.729655 26070 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:40.732921 26070 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:40.802139 26070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb09ee5859d648e9816844e4a6a218f3: Generated new TSK 0
I20260619 18:46:40.803815 26070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:46:40.847872 26094 catalog_manager.cc:1259] Loaded cluster ID: 2af94d4d99ba4519aabda7a37b457344
I20260619 18:46:40.848076 26094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: loading cluster ID for follower catalog manager: success
I20260619 18:46:40.852525 26094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: acquiring CA information for follower catalog manager: success
I20260619 18:46:40.867835 26094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edc0d23ea86d4ebcb60a017cba91830a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:46:40.958763 26034 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:40.927715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.3.85.190:34599)
0619 18:46:40.927966 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.927974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.952273 (+ 24299us) server_negotiation.cc:407] Connection header received
0619 18:46:40.954731 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.954738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.954843 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.954925 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.958478 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48962: BlockingRecv error: recv got EOF from 127.0.0.1:48962 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 18:46:40.961061 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:40.945009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.3.85.188:45725)
0619 18:46:40.945263 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.945272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.957385 (+ 12113us) server_negotiation.cc:407] Connection header received
0619 18:46:40.959340 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.959347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.959481 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.959560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.960774 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39900: BlockingRecv error: recv got EOF from 127.0.0.1:39900 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 18:46:40.962373 25960 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:40.933886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.3.85.189:33471)
0619 18:46:40.934133 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.934140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:40.954138 (+ 19998us) server_negotiation.cc:407] Connection header received
0619 18:46:40.956475 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:40.956482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:40.956597 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:40.956682 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:40.962106 (+ 5424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45384: BlockingRecv error: recv got EOF from 127.0.0.1:45384 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:46:41.013334 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:40.996173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.3.85.189:33471)
0619 18:46:40.996400 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:40.996414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.004790 (+ 8376us) server_negotiation.cc:407] Connection header received
0619 18:46:41.005367 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.005374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.005611 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.005781 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.006516 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.007744 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.009966 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.010735 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.010873 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.011685 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.012432 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.012559 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.012842 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.012960 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 18:46:41.071310 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.068235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.3.85.188:45725)
0619 18:46:41.068435 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.068442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.069109 (+ 667us) server_negotiation.cc:407] Connection header received
0619 18:46:41.069518 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.069542 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.069681 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.069765 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.071015 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39910: BlockingRecv error: recv got EOF from 127.0.0.1:39910 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:46:41.134214 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.114928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.3.85.189:33471)
0619 18:46:41.115158 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.115164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.127168 (+ 12004us) server_negotiation.cc:407] Connection header received
0619 18:46:41.127393 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.127400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.127521 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.127631 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.128379 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.129554 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.131717 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.132422 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.132533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.133250 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.133414 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.133491 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.133735 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.133845 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:46:41.202405 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.176773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.3.85.188:45725)
0619 18:46:41.177019 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.177029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.200299 (+ 23270us) server_negotiation.cc:407] Connection header received
0619 18:46:41.200837 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.200844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.200934 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.201008 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.202128 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39912: BlockingRecv error: recv got EOF from 127.0.0.1:39912 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:46:41.240011 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.231970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.3.85.189:33471)
0619 18:46:41.232171 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.232177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.232763 (+ 586us) server_negotiation.cc:407] Connection header received
0619 18:46:41.232995 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.233003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.233115 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.233233 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.233781 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.235217 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.237441 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.238261 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.238434 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.239035 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.239211 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.239321 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.239501 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.239606 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:46:41.251624 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.249407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.3.85.188:45725)
0619 18:46:41.249620 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.249627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.250119 (+ 492us) server_negotiation.cc:407] Connection header received
0619 18:46:41.250300 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.250306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.250401 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.250544 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.251372 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39926: BlockingRecv error: recv got EOF from 127.0.0.1:39926 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:46:41.265586 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.259721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.3.85.189:33471)
0619 18:46:41.260006 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.260014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.260398 (+ 384us) server_negotiation.cc:407] Connection header received
0619 18:46:41.260647 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.260652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.260731 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.260836 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.261283 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.262094 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.263727 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.264223 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.264371 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.264796 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.264915 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.265044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.265207 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.265299 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260619 18:46:41.273960 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.271699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.3.85.188:45725)
0619 18:46:41.272010 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.272017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.272417 (+ 400us) server_negotiation.cc:407] Connection header received
0619 18:46:41.272594 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.272615 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.272693 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.272789 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.273718 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39932: BlockingRecv error: recv got EOF from 127.0.0.1:39932 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260619 18:46:41.294219 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.286508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.3.85.189:33471)
0619 18:46:41.286809 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.286822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.287195 (+ 373us) server_negotiation.cc:407] Connection header received
0619 18:46:41.287415 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.287420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.287497 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.287585 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.288185 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.289104 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.291882 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.292433 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.292593 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.293146 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.293289 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.293422 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.293649 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.293739 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 18:46:41.313647 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.308811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.3.85.188:45725)
0619 18:46:41.309045 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.309054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.311682 (+ 2628us) server_negotiation.cc:407] Connection header received
0619 18:46:41.312070 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.312078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.312185 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.312270 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.313405 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39940: BlockingRecv error: recv got EOF from 127.0.0.1:39940 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:46:41.340873 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.334442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.3.85.189:33471)
0619 18:46:41.334755 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.334767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.335148 (+ 381us) server_negotiation.cc:407] Connection header received
0619 18:46:41.335340 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.335344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.335422 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.335507 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.336067 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.336909 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.338984 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.339423 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.339547 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.340093 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.340221 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.340296 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.340491 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.340574 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 18:46:41.350286 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.347832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.3.85.188:45725)
0619 18:46:41.348219 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.348229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.348821 (+ 592us) server_negotiation.cc:407] Connection header received
0619 18:46:41.348958 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.348964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.349067 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.349202 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.350020 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39952: BlockingRecv error: recv got EOF from 127.0.0.1:39952 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 18:46:41.395875 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.388629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.3.85.189:33471)
0619 18:46:41.388880 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.388888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.389130 (+ 242us) server_negotiation.cc:407] Connection header received
0619 18:46:41.389387 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.389392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.389467 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.389563 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.390114 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.390959 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.393176 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.393881 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.394009 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.394644 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.394818 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.394941 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.395172 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.395466 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 18:46:41.410941 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.408641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.3.85.188:45725)
0619 18:46:41.408888 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.408896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.409292 (+ 396us) server_negotiation.cc:407] Connection header received
0619 18:46:41.409536 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.409542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.409634 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.409728 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.410694 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39956: BlockingRecv error: recv got EOF from 127.0.0.1:39956 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:46:41.484318 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.478637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.3.85.189:33471)
0619 18:46:41.478815 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.478819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.479313 (+ 494us) server_negotiation.cc:407] Connection header received
0619 18:46:41.479478 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.479482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.479554 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.479645 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.480111 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.480841 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.482322 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.482858 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.482973 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.483491 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.483636 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.483718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.483999 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.484077 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:46:41.492479 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:41.490438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.3.85.188:45725)
0619 18:46:41.490756 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.490763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.490992 (+ 229us) server_negotiation.cc:407] Connection header received
0619 18:46:41.491258 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.491264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.491350 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.491463 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:41.492239 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39968: BlockingRecv error: recv got EOF from 127.0.0.1:39968 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:46:41.562566 26071 catalog_manager.cc:1259] Loaded cluster ID: 2af94d4d99ba4519aabda7a37b457344
I20260619 18:46:41.562680 26071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: loading cluster ID for follower catalog manager: success
I20260619 18:46:41.565351 26071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: acquiring CA information for follower catalog manager: success
I20260619 18:46:41.567422 26071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5adc4d636d74ec0b329d67dc0ac4d1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:46:41.629663 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.624138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.3.85.189:33471)
0619 18:46:41.624341 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.624346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.624668 (+ 322us) server_negotiation.cc:407] Connection header received
0619 18:46:41.625149 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.625154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.625238 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.625323 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.625710 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.626427 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.627959 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.628415 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.628528 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.628912 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.629034 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.629116 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.629361 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.629439 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:46:41.643580 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.637336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39970 (local address 127.3.85.188:45725)
0619 18:46:41.637535 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.637539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.638018 (+ 479us) server_negotiation.cc:407] Connection header received
0619 18:46:41.638172 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.638177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.638344 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.638525 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.638952 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.639671 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.641102 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.641528 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.641665 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.642250 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.642968 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.643097 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.643253 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.643331 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:46:41.659902 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.650706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.3.85.190:34599)
0619 18:46:41.652139 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.652147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.652168 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:46:41.652302 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.652310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.652574 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.652775 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.653535 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.654752 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.656890 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.657424 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.657592 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.658419 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.658994 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.659112 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.659281 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.659384 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":1162,"threads_started":1}
I20260619 18:46:41.660753 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.651263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.3.85.189:33471)
0619 18:46:41.651506 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.651515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.653609 (+ 2094us) server_negotiation.cc:407] Connection header received
0619 18:46:41.653961 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.653970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.654098 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.654172 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.655151 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.656117 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.658358 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.659024 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.659162 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.659829 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.659971 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.660101 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.660394 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.660491 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:46:41.661607 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:41.652431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.3.85.188:45725)
0619 18:46:41.652693 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:41.652703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:41.654063 (+ 1360us) server_negotiation.cc:407] Connection header received
0619 18:46:41.654327 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:41.654333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:41.654429 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:41.654528 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:41.655461 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.656576 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.658718 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:41.659335 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:41.659463 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:41.660093 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:41.660233 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:41.660358 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:41.661235 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 18:46:41.661325 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:46:41.666169 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25813
I20260619 18:46:41.667476 25875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:41.783162 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25813
I20260619 18:46:41.806097 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25887
I20260619 18:46:41.807374 25949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:41.917769 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25887
I20260619 18:46:41.938880 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 25962
I20260619 18:46:41.940186 26024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:42.051833 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 25962
2026-06-19T18:46:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4834 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:42 dist-test-slave-09fp krb5kdc[26208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:42 dist-test-slave-09fp krb5kdc[26208](info): set up 2 sockets
Jun 19 18:46:42 dist-test-slave-09fp krb5kdc[26208](info): commencing operation
krb5kdc: starting...
W20260619 18:46:43.203238 3414 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.011s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/test-user.keytab.
Jun 19 18:46:43 dist-test-slave-09fp krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.85.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
Entry for principal kudu/127.3.85.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab.
I20260619 18:46:43.491092 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39271
--webserver_interface=127.3.85.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:43.869031 26221 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:43.869415 26221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:43.869488 26221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:43.869581 26221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:43.877552 26221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:43.877686 26221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:43.877753 26221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:43.877804 26221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:43.877878 26221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:43.888636 26221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.190.keytab
--principal=kudu/127.3.85.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.190:39271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.85.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:43.890154 26221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:43.891690 26221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:43.905180 26226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:43.905213 26227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:43.905654 26229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:43.905920 26221 server_base.cc:1034] running on GCE node
I20260619 18:46:43.906900 26221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:43.908510 26221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:43.909772 26221 hybrid_clock.cc:648] HybridClock initialized: now 1781894803909698 us; error 40 us; skew 500 ppm
I20260619 18:46:43.910190 26221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:43 dist-test-slave-09fp krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:43.913878 26221 init.cc:377] Logged in from keytab as kudu/127.3.85.190@KRBTEST.COM (short username kudu)
I20260619 18:46:43.915735 26221 webserver.cc:460] Webserver started at http://127.3.85.190:43183/ using document root <none> and password file <none>
I20260619 18:46:43.916575 26221 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:43.916741 26221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:43.917099 26221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:43.920434 26221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data/instance:
uuid: "8611ceee130f46e1b358817b2dfc5bd8"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-09fp"
server_key: "9c59424cc50cbccd84d21e736b723e59"
server_key_iv: "e0cd6995387bebc0bff25ecca8bb42f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.921336 26221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance:
uuid: "8611ceee130f46e1b358817b2dfc5bd8"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-09fp"
server_key: "9c59424cc50cbccd84d21e736b723e59"
server_key_iv: "e0cd6995387bebc0bff25ecca8bb42f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.927322 26221 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:46:43.931427 26236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:43.933092 26221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:46:43.933338 26221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal
uuid: "8611ceee130f46e1b358817b2dfc5bd8"
format_stamp: "Formatted at 2026-06-19 18:46:43 on dist-test-slave-09fp"
server_key: "9c59424cc50cbccd84d21e736b723e59"
server_key_iv: "e0cd6995387bebc0bff25ecca8bb42f6"
server_key_version: "encryptionkey@0"
I20260619 18:46:43.933600 26221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:43.948025 26221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:43.953030 26221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:43.953359 26221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:43.973613 26221 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.190:39271
I20260619 18:46:43.973634 26287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.190:39271 every 8 connection(s)
I20260619 18:46:43.975170 26221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:43.979429 26288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:43.980791 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 26221
I20260619 18:46:43.981005 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-0/wal/instance
I20260619 18:46:43.981482 3414 external_mini_cluster.cc:1428] Setting key b6736866ef2696e7aef8345941581473
I20260619 18:46:43.986863 26288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:44 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.189@KRBTEST.COM" created.
I20260619 18:46:44.016269 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:43.990573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.3.85.190:39271)
0619 18:46:43.991639 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:43.991673 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:43.992194 (+ 521us) server_negotiation.cc:407] Connection header received
0619 18:46:43.995824 (+ 3630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:43.995831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:43.996154 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:43.996343 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:43.998239 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:43.999668 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.002222 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.002950 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.003123 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.007919 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.007928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.007943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:44.007978 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.011251 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.012234 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.012244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.012250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.012422 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.012875 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.012882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.012885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.013450 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.013686 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.014269 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.014687 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":462,"threads_started":1}
I20260619 18:46:44.016304 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:43.990573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39271 (local address 127.0.0.1:35976)
0619 18:46:43.991978 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:43.992036 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:43.995258 (+ 3222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:43.996467 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:43.996481 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:43.997403 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:43.997963 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:43.997995 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:43.999862 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:43.999870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.002005 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.002031 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.003257 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.003268 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.003514 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.004384 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:44.004430 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:44.007610 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:44.011426 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.011452 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.011470 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.011967 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.012568 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.012573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.012576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.012758 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.013611 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:44.013621 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:44.013873 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 18:46:44.014147 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 18:46:44.014820 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":182,"spinlock_wait_cycles":72960,"thread_start_us":831,"threads_started":1}
I20260619 18:46:44.020105 26288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:44.021488 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.021065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36714)
0619 18:46:44.021245 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.021360 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 18:46:44.021801 26238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
I20260619 18:46:44.023501 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.023116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36716)
0619 18:46:44.023282 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.023350 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:46:44.023721 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:35677: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
Entry for principal kudu/127.3.85.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab.
I20260619 18:46:44.047323 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:35677
--webserver_interface=127.3.85.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:44.073105 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } attempt: 1
I20260619 18:46:44.074340 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.073861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36726)
0619 18:46:44.074084 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.074203 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:46:44.075657 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.075083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36732)
0619 18:46:44.075417 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.075503 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:46:44.075896 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:35677: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
I20260619 18:46:44.144207 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } attempt: 2
I20260619 18:46:44.145444 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.144959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36740)
0619 18:46:44.145172 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.145293 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:46:44.146564 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.146110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36754)
0619 18:46:44.146294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.146361 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:46:44.146782 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:35677: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
I20260619 18:46:44.238112 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } attempt: 3
I20260619 18:46:44.239638 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.239040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36764)
0619 18:46:44.239367 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.239482 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 18:46:44.241118 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.240653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36768)
0619 18:46:44.240861 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.240945 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 18:46:44.241375 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:35677: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
I20260619 18:46:44.384653 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } attempt: 4
I20260619 18:46:44.386013 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.385397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36780)
0619 18:46:44.385785 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.385877 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260619 18:46:44.387105 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.386736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36792)
0619 18:46:44.386890 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.386972 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:46:44.387295 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.189:35677: Network error: Client connection negotiation failed: client connection to 127.3.85.189:35677: connect: Connection refused (error 111)
W20260619 18:46:44.413594 26295 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:44.414053 26295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:44.414186 26295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:44.414330 26295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:44.422734 26295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:44.422910 26295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:44.423024 26295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:44.423118 26295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:44.423209 26295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:44.435207 26295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.189.keytab
--principal=kudu/127.3.85.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.189:35677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.85.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:44.436669 26295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:44.438112 26295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:44.453531 26300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:44.453440 26301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:44.453495 26303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:44.454190 26295 server_base.cc:1034] running on GCE node
I20260619 18:46:44.455255 26295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:44.457226 26295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:44.458767 26295 hybrid_clock.cc:648] HybridClock initialized: now 1781894804458631 us; error 81 us; skew 500 ppm
I20260619 18:46:44.459194 26295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:44 dist-test-slave-09fp krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:44.463220 26295 init.cc:377] Logged in from keytab as kudu/127.3.85.189@KRBTEST.COM (short username kudu)
I20260619 18:46:44.465493 26295 webserver.cc:460] Webserver started at http://127.3.85.189:40273/ using document root <none> and password file <none>
I20260619 18:46:44.466337 26295 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:44.466516 26295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:44.466881 26295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:44.470748 26295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data/instance:
uuid: "165c6c45dfa0452485917505c0a80843"
format_stamp: "Formatted at 2026-06-19 18:46:44 on dist-test-slave-09fp"
server_key: "1379fd29bc11c1b02cccbd60fd6c34cf"
server_key_iv: "6ee039cbd19cbe340332c2bfd01b5a20"
server_key_version: "encryptionkey@0"
I20260619 18:46:44.471740 26295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal/instance:
uuid: "165c6c45dfa0452485917505c0a80843"
format_stamp: "Formatted at 2026-06-19 18:46:44 on dist-test-slave-09fp"
server_key: "1379fd29bc11c1b02cccbd60fd6c34cf"
server_key_iv: "6ee039cbd19cbe340332c2bfd01b5a20"
server_key_version: "encryptionkey@0"
I20260619 18:46:44.479101 26295 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 18:46:44.484263 26310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:44.486366 26295 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:46:44.486806 26295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal
uuid: "165c6c45dfa0452485917505c0a80843"
format_stamp: "Formatted at 2026-06-19 18:46:44 on dist-test-slave-09fp"
server_key: "1379fd29bc11c1b02cccbd60fd6c34cf"
server_key_iv: "6ee039cbd19cbe340332c2bfd01b5a20"
server_key_version: "encryptionkey@0"
I20260619 18:46:44.487113 26295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:44.504084 26295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:44.509668 26295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:44.510041 26295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:44.534176 26295 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.189:35677
I20260619 18:46:44.534195 26361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.189:35677 every 8 connection(s)
I20260619 18:46:44.535849 26295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/data/info.pb
I20260619 18:46:44.537942 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 26295
I20260619 18:46:44.538182 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-1/wal/instance
I20260619 18:46:44.538779 3414 external_mini_cluster.cc:1428] Setting key 3953d703963beb9a06e6974ad7461ee5
I20260619 18:46:44.540010 26362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:44.547201 26362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:44 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.85.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.85.188@KRBTEST.COM" created.
I20260619 18:46:44.570757 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.550239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.3.85.190:39271)
0619 18:46:44.550861 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:44.550866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:44.552040 (+ 1174us) server_negotiation.cc:407] Connection header received
0619 18:46:44.554250 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:44.554257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:44.554365 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:44.554527 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:44.556403 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.557136 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.558941 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.559375 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.559536 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.564206 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.564212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.564215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:44.564265 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.567025 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.568026 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.568031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.568033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.568165 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.568663 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.568667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.568669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.568968 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.569136 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.569923 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.570298 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":384,"threads_started":1}
I20260619 18:46:44.571847 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.550924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39271 (local address 127.0.0.1:35980)
0619 18:46:44.551830 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.551873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:44.553829 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:44.554834 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:44.554845 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:44.555475 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:44.556120 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.556141 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.557349 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.557353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.558760 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.558776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.559645 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.559654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.559921 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.560794 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:44.560844 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:44.563884 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:44.567225 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.567237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.567261 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.567864 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.568294 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.568299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.568302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.568524 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.569113 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:44.569122 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:44.569520 (+ 398us) client_negotiation.cc:769] Sending connection context
0619 18:46:44.569810 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 18:46:44.570578 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":390,"threads_started":1}
I20260619 18:46:44.573994 26362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:44 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:44.593580 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.575118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36800)
0619 18:46:44.575581 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.575606 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:44.575832 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:44.576413 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:44.576424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:44.577143 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:44.577377 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.577392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.579838 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.579843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.580727 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.580741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.582108 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.582113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.582252 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.583080 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:44.583102 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:44.585676 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:44.590697 (+ 5021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.590701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.590715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.591118 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.591603 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.591607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.591609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.591771 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.592662 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:44.592665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:44.592849 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 18:46:44.592970 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 18:46:44.593307 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260619 18:46:44.594051 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.575238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.3.85.189:35677)
0619 18:46:44.575940 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:44.575947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:44.575975 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:46:44.576024 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:44.576030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:44.576239 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:44.576365 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:44.577612 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.579645 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.580898 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.581922 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.582079 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.585964 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.585988 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.586012 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:44.586066 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.590497 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.591268 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.591282 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.591289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.591458 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.591887 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.591895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.591902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.592490 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.592739 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.593218 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.593522 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":325,"threads_started":1}
I20260619 18:46:44.597208 26362 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:44.598770 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.598084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60874)
0619 18:46:44.598309 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.598552 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 18:46:44.599174 26313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.601087 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.600636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60878)
0619 18:46:44.600859 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.600958 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 18:46:44.601347 26362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
Entry for principal kudu/127.3.85.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab.
I20260619 18:46:44.613713 3414 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
/tmp/dist-test-taskHiUM6J/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:41897
--webserver_interface=127.3.85.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.85.148:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:46:44.650665 26362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 1
I20260619 18:46:44.651845 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.651367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60880)
0619 18:46:44.651575 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.651685 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 18:46:44.652863 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.652499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60884)
0619 18:46:44.652672 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.652753 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 18:46:44.653076 26362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.686587 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } attempt: 5
Jun 19 18:46:44 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:44.702740 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.687356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36806)
0619 18:46:44.687770 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.687798 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:44.687959 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:44.688427 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:44.688433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:44.688952 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:44.689185 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.689197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.690262 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.690268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.691768 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:44.691792 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.692788 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.692795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:44.693040 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.693794 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:44.693819 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:44.696410 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:44.699774 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.699780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.699785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.700400 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.700863 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:44.700868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:44.700872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:44.701025 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:44.701606 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:44.701610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:44.701779 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 18:46:44.701959 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 18:46:44.702296 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260619 18:46:44.702740 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.687639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.3.85.189:35677)
0619 18:46:44.687805 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:44.687834 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:44.687870 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:46:44.688127 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:44.688133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:44.688240 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:44.688356 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:44.689363 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.690093 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.692011 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:44.692591 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:44.692719 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:44.696692 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:44.696699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:44.696702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:44.696742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:44.699565 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.700561 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.700574 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.700577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.700726 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:44.701149 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:44.701154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:44.701159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:44.701470 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:44.701643 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:44.702068 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 18:46:44.702303 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:46:44.703877 26288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:44.705097 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.704634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60890)
0619 18:46:44.704823 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.704920 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 18:46:44.706202 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.705783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60892)
0619 18:46:44.705929 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.706047 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 18:46:44.706601 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.721320 26362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 2
I20260619 18:46:44.722781 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.722121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60902)
0619 18:46:44.722409 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.722602 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260619 18:46:44.723963 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.723538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60914)
0619 18:46:44.723716 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.723820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 18:46:44.724229 26362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.757936 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 1
I20260619 18:46:44.759377 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.758757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60920)
0619 18:46:44.758990 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.759118 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:46:44.760761 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.760312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60934)
0619 18:46:44.760490 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.760592 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:46:44.761092 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.815479 26362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 3
I20260619 18:46:44.816740 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.816239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60946)
0619 18:46:44.816463 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.816578 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 18:46:44.817993 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.817525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60952)
0619 18:46:44.817779 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.817865 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260619 18:46:44.818271 26362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.829363 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 2
I20260619 18:46:44.830792 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.830197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60962)
0619 18:46:44.830494 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.830606 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:46:44.832026 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.831640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60964)
0619 18:46:44.831825 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.831907 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 18:46:44.832298 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.938536 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 3
I20260619 18:46:44.939815 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.939245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60972)
0619 18:46:44.939465 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.939557 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 18:46:44.941006 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.940588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60982)
0619 18:46:44.940795 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.940883 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260619 18:46:44.941231 26288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
I20260619 18:46:44.961530 26362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 4
I20260619 18:46:44.962781 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.962213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:60996)
0619 18:46:44.962524 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.962616 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 18:46:44.963858 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:44.963501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:32768)
0619 18:46:44.963659 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:44.963743 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:46:44.964095 26362 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.85.188:41897: Network error: Client connection negotiation failed: client connection to 127.3.85.188:41897: connect: Connection refused (error 111)
W20260619 18:46:44.979656 26370 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:46:44.980069 26370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:44.980218 26370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:44.980365 26370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:44.988226 26370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:44.988436 26370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:44.988575 26370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:44.988663 26370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:44.988775 26370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:44.999779 26370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.85.148:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.85.190:39271,127.3.85.189:35677,127.3.85.188:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/krb5kdc/kudu_127.3.85.188.keytab
--principal=kudu/127.3.85.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.85.188:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.85.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:45.001300 26370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:45.002761 26370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:45.015513 26378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:45.015594 26376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:45.015482 26375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:45.020421 26370 server_base.cc:1034] running on GCE node
I20260619 18:46:45.021385 26370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:45.023336 26370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:45.024654 26370 hybrid_clock.cc:648] HybridClock initialized: now 1781894805024567 us; error 67 us; skew 500 ppm
I20260619 18:46:45.025113 26370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:45.028774 26370 init.cc:377] Logged in from keytab as kudu/127.3.85.188@KRBTEST.COM (short username kudu)
I20260619 18:46:45.030694 26370 webserver.cc:460] Webserver started at http://127.3.85.188:38277/ using document root <none> and password file <none>
I20260619 18:46:45.031553 26370 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:45.031698 26370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:45.032042 26370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:45.035473 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data/instance:
uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-09fp"
server_key: "b3f937487b1b0ff35e4364853f732558"
server_key_iv: "c7a3fc53356ede4eedbd9de8be2ac9c1"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.036491 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal/instance:
uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-09fp"
server_key: "b3f937487b1b0ff35e4364853f732558"
server_key_iv: "c7a3fc53356ede4eedbd9de8be2ac9c1"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.042935 26370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 18:46:45.047693 26385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:45.049275 26370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:46:45.049612 26370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data,/tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal
uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba"
format_stamp: "Formatted at 2026-06-19 18:46:45 on dist-test-slave-09fp"
server_key: "b3f937487b1b0ff35e4364853f732558"
server_key_iv: "c7a3fc53356ede4eedbd9de8be2ac9c1"
server_key_version: "encryptionkey@0"
I20260619 18:46:45.049893 26370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:45.064273 26370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:45.068908 26370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:45.069218 26370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:45.090404 26370 rpc_server.cc:307] RPC server started. Bound to: 127.3.85.188:41897
I20260619 18:46:45.090423 26436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.85.188:41897 every 8 connection(s)
I20260619 18:46:45.092074 26370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/data/info.pb
I20260619 18:46:45.093406 3414 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu as pid 26370
I20260619 18:46:45.093627 3414 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHiUM6J/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894429649633-3414-0/minicluster-data/master-2/wal/instance
I20260619 18:46:45.094092 3414 external_mini_cluster.cc:1428] Setting key 99d31d62513125d974694eaf15590f72
I20260619 18:46:45.096935 26437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:45.104305 26437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
I20260619 18:46:45.118502 26288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 4
I20260619 18:46:45.121467 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.095990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.3.85.188:41897)
0619 18:46:45.096881 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.096894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.096946 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:46:45.098966 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.098992 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.099502 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.099998 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.101928 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.103501 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.105011 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.105913 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.106165 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.111782 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.111792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.111806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.111853 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.115714 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.116620 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.116630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.116649 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.116822 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.117282 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.117287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.117289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.118138 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.118505 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.119267 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.119769 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":402,"threads_started":1}
I20260619 18:46:45.123782 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.107107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.3.85.190:39271)
0619 18:46:45.107701 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.107709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.108118 (+ 409us) server_negotiation.cc:407] Connection header received
0619 18:46:45.108331 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.108340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.108453 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.108592 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.109767 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.110800 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.112316 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.112748 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.112906 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.116588 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.116594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.116597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.116635 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.120277 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.121253 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.121258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.121262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.121450 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.121951 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.121956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.121959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.122317 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.122501 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.123041 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.123315 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":292,"threads_started":1}
I20260619 18:46:45.123876 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.107084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.190:39271 (local address 127.0.0.1:35986)
0619 18:46:45.107941 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:45.107993 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:45.108180 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:45.108674 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:45.108679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:45.109310 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:45.109574 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.109584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.110985 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.110989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.112112 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.112131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.112955 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.112962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.113186 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.113876 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:45.113904 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:45.116348 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:45.120453 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.120468 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.120477 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.121081 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.121597 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.121601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.121605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.121814 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.122514 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:45.122522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:45.122711 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 18:46:45.122930 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 18:46:45.123312 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":438,"threads_started":1}
I20260619 18:46:45.125538 26437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.190@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.190@KRBTEST.COM
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:45.142045 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.119270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:32800)
0619 18:46:45.119822 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:45.119846 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:45.119976 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:45.121035 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:45.121044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:45.121556 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:45.121807 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.121820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.125425 (+ 3605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.125433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.126791 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.126805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.127769 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.127775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.127995 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.129514 (+ 1519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:45.129541 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:45.132806 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:45.136931 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.136937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.136941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.137638 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.138192 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.138197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.138199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.138428 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.139124 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:45.139132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:45.141136 (+ 2004us) client_negotiation.cc:769] Sending connection context
0619 18:46:45.141294 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 18:46:45.141626 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397}
I20260619 18:46:45.142431 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.119847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.3.85.188:41897)
0619 18:46:45.120733 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.120738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.120758 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:46:45.120802 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.120806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.120863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.120993 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.122594 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.125249 (+ 2655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.126932 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.127618 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.127746 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.133087 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.133094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.133098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.133148 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.136780 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.137818 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.137827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.137833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.138005 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.138564 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.138570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.138573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.138962 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.139124 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.141442 (+ 2318us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.141730 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":528,"threads_started":1}
I20260619 18:46:45.144604 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.125172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.3.85.190:39271)
0619 18:46:45.125347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.125352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.125371 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:46:45.125440 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.125444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.125550 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.125691 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.127465 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.128704 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.130063 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.130657 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.130856 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.136589 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.136598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.136606 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.136669 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.140765 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.141924 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.141930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.141934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.142142 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.142748 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.142754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.142758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.143180 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.143514 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.143921 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.144198 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:46:45.144635 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.126663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.3.85.189:35677)
0619 18:46:45.126872 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.126881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.127167 (+ 286us) server_negotiation.cc:407] Connection header received
0619 18:46:45.127345 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.127351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.127433 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.127538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.128580 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.129229 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.130991 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.131680 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.131778 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.136771 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.136782 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.136791 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.136843 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.140851 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.141948 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.141966 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.141970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.142148 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.142790 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.142796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.142799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.143186 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.143398 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.143979 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.144243 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:46:45.145627 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.126301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.189:35677 (local address 127.0.0.1:36812)
0619 18:46:45.127041 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:45.127062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:45.127211 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:45.127598 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:45.127605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:45.128168 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:45.128361 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.128374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.129390 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.129395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.130805 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.130823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.132032 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.132037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.132199 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.132758 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:45.132776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:45.136329 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:45.141058 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.141063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.141067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.141761 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.142330 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.142334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.142338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.142591 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.143471 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:45.143476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:45.143676 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 18:46:45.144923 (+ 1247us) client_negotiation.cc:240] Negotiation successful
0619 18:46:45.145252 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602}
I20260619 18:46:45.147440 26437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:46:45.149266 26288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: Bootstrap starting.
I20260619 18:46:45.153286 26288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:45.154871 26288 log.cc:824] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.188@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.85.189@KRBTEST.COM
I20260619 18:46:45.158718 26288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: No bootstrap required, opened a new log
I20260619 18:46:45.165203 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.148375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:32816)
0619 18:46:45.149095 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:45.149117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:45.149230 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:45.149585 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:45.149589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:45.150031 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:45.150175 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.150184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.151686 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.151691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.152787 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.152801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.153890 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.153896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.154112 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.154707 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:45.154738 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:45.158687 (+ 3949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:45.162152 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.162161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.162167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.162669 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.163215 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.163222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.163226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.163413 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.164025 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:45.164030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:45.164210 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 18:46:45.164376 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:46:45.164735 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625}
I20260619 18:46:45.165390 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.148726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.3.85.188:41897)
0619 18:46:45.148981 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.148987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.149322 (+ 335us) server_negotiation.cc:407] Connection header received
0619 18:46:45.149353 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.149356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.149465 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.149576 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.150345 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.151519 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.152951 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.153723 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.153885 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.159019 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.159025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.159029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.159063 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.161973 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.162802 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.162810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.162818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.162981 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.163566 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.163574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.163579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.163885 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.164086 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.164511 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.164954 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":134}
I20260619 18:46:45.166782 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.147074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.3.85.189:35677)
0619 18:46:45.147248 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.147253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.147272 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:46:45.147323 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.147328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.147401 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.147504 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.149262 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.150381 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.152262 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.152933 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.153094 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.159075 (+ 5981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.159083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.159087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.159132 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.162838 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.163647 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.163653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.163656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.163833 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.164323 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.164329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.164334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.164749 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.165039 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.166119 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.166386 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:46:45.170397 26288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.170877 26288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8611ceee130f46e1b358817b2dfc5bd8, State: Initialized, Role: FOLLOWER
I20260619 18:46:45.171066 26437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: Bootstrap starting.
I20260619 18:46:45.171442 26288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.172473 3414 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:46:45.172593 26445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.173264 26445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.173971 26288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:45.175450 26437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:45.176060 26263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8611ceee130f46e1b358817b2dfc5bd8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36000
I20260619 18:46:45.176522 26263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:46:45.176748 26263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:45.176978 26437 log.cc:824] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:45.180147 26263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.181059 26437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: No bootstrap required, opened a new log
I20260619 18:46:45.182479 26263 leader_election.cc:288] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 165c6c45dfa0452485917505c0a80843 (127.3.85.189:35677), 8577dcdb254f4b55b9b8dfa98d4c9aba (127.3.85.188:41897)
W20260619 18:46:45.206950 26242 tablet.cc:2369] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:45.194285 26221 master_runner.cc:186] Error getting master registration for 127.3.85.189:35677: OK, instance_id { permanent_uuid: "165c6c45dfa0452485917505c0a80843" instance_seqno: 1781894804528421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:46:45.210271 26221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:45.212883 26437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
W20260619 18:46:45.213512 26459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:46:45.213985 26459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:46:45.213709 26437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8577dcdb254f4b55b9b8dfa98d4c9aba, State: Initialized, Role: FOLLOWER
I20260619 18:46:45.224931 26437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.232789 26466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.234169 26466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.242635 26412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8611ceee130f46e1b358817b2dfc5bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba"
I20260619 18:46:45.243356 26412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:45.247727 26412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8611ceee130f46e1b358817b2dfc5bd8 in term 1.
I20260619 18:46:45.248644 26239 leader_election.cc:302] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8577dcdb254f4b55b9b8dfa98d4c9aba, 8611ceee130f46e1b358817b2dfc5bd8; no voters:
I20260619 18:46:45.249260 26445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:45.251871 26437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:45.261368 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.214628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.3.85.188:41897)
0619 18:46:45.215168 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.215177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.231824 (+ 16647us) server_negotiation.cc:407] Connection header received
0619 18:46:45.232205 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.232211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.232312 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.232393 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.235558 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.237070 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.240453 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.242904 (+ 2451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.243054 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.254093 (+ 11039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.254101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.254106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.254153 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.257599 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.258588 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.258596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.258600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.258793 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.259334 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.259339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.259342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.259754 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.260060 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.260684 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.260953 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260619 18:46:45.266949 26362 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } attempt: 5
I20260619 18:46:45.269495 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.214868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.3.85.190:39271)
0619 18:46:45.215063 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.215070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.236554 (+ 21484us) server_negotiation.cc:407] Connection header received
0619 18:46:45.237218 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.237225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.237334 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.237418 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.247152 (+ 9734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.248369 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.250659 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.251501 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.251632 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.260922 (+ 9290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.260930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.260934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.260981 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.265142 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.266165 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.266173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.266180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.266378 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.266898 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.266903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.266907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.267321 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.267586 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.268750 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.269074 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:46:45.271142 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.215343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.3.85.189:35677)
0619 18:46:45.215558 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.215566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.222627 (+ 7061us) server_negotiation.cc:407] Connection header received
0619 18:46:45.223152 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.223171 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.223266 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.223342 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.226417 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.227801 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.231071 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.231783 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.231933 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.251887 (+ 19954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.251896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.251901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.251961 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.257339 (+ 5378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.264794 (+ 7455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.264804 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.264810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.265024 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.266591 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.266599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.266604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.267110 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.267400 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.269397 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.270754 (+ 1357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:46:45.276746 26445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 8611ceee130f46e1b358817b2dfc5bd8, State: Running, Role: LEADER
I20260619 18:46:45.278003 26445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.287808 26467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8611ceee130f46e1b358817b2dfc5bd8. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.289062 26467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: This master's current role is: LEADER
Jun 19 18:46:45 dist-test-slave-09fp krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.85.189@KRBTEST.COM for kudu/127.3.85.188@KRBTEST.COM
I20260619 18:46:45.306437 26474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:45.309983 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.268219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.85.188:41897 (local address 127.0.0.1:32848)
0619 18:46:45.269764 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:45.269784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:45.269913 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:45.280235 (+ 10322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:45.280245 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:45.280854 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:45.281091 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.281106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.282882 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.282892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.284642 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:45.284665 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.286940 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.286949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:45.287236 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.290017 (+ 2781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:45.290066 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:45.297118 (+ 7052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:45.306482 (+ 9364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.306488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.306495 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.307198 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.307760 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:45.307765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:45.307768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:45.307964 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:45.308817 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:45.308823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:45.309018 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 18:46:45.309209 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 18:46:45.309558 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392}
I20260619 18:46:45.312254 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:45.270190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.3.85.188:41897)
0619 18:46:45.270375 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.270380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.270402 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:46:45.279492 (+ 9090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.279501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.279619 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.279768 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.281308 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.282470 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.284867 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:45.286480 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:45.286651 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:45.297680 (+ 11029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:45.297688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:45.297694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:45.297759 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:45.306186 (+ 8427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.307398 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.307405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.307410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.307595 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:45.308118 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:45.308125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:45.308130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:45.308565 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:45.308727 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:45.310438 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0619 18:46:45.311700 (+ 1262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:46:45.313469 26474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 18:46:45.321741 26391 tablet.cc:2369] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:45.322504 26362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: Bootstrap starting.
I20260619 18:46:45.328140 26362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:45.333778 26362 log.cc:824] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:45.342402 26362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: No bootstrap required, opened a new log
I20260619 18:46:45.347568 26412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [term 1 FOLLOWER]: Refusing update from remote peer 8611ceee130f46e1b358817b2dfc5bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:45.348959 26467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:45.362226 26362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.363194 26362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165c6c45dfa0452485917505c0a80843, State: Initialized, Role: FOLLOWER
I20260619 18:46:45.364207 26362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } }
I20260619 18:46:45.375847 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.376802 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.375985 26337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8611ceee130f46e1b358817b2dfc5bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "165c6c45dfa0452485917505c0a80843"
I20260619 18:46:45.376315 26336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:45.381494 26336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [term 1 FOLLOWER]: Refusing update from remote peer 8611ceee130f46e1b358817b2dfc5bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:46:45.382596 26467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:46:45.385370 26337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8611ceee130f46e1b358817b2dfc5bd8 in term 1.
I20260619 18:46:45.375936 26362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:45.405249 26445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.405623 26445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:45.406216 26486 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:46:45.406644 26466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8611ceee130f46e1b358817b2dfc5bd8. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.406584 26487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.406980 26466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.407263 26474 catalog_manager.cc:1347] Generated new cluster ID: 21d820f5482e445fa1120d97d1f1a724
I20260619 18:46:45.407366 26474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:45.407001 26487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.414253 26370 master_runner.cc:186] Error getting master registration for 127.3.85.189:35677: OK, instance_id { permanent_uuid: "165c6c45dfa0452485917505c0a80843" instance_seqno: 1781894804528421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:46:45.425966 26370 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:46:45.426103 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8611ceee130f46e1b358817b2dfc5bd8. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.426465 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.430290 26467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.430612 26467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:45.440560 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8611ceee130f46e1b358817b2dfc5bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8611ceee130f46e1b358817b2dfc5bd8" member_type: VOTER last_known_addr { host: "127.3.85.190" port: 39271 } } peers { permanent_uuid: "165c6c45dfa0452485917505c0a80843" member_type: VOTER last_known_addr { host: "127.3.85.189" port: 35677 } } peers { permanent_uuid: "8577dcdb254f4b55b9b8dfa98d4c9aba" member_type: VOTER last_known_addr { host: "127.3.85.188" port: 41897 } } }
I20260619 18:46:45.440893 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:46:45.449149 26474 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:45.451262 26474 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:45.473901 26503 catalog_manager.cc:1259] Loaded cluster ID: 21d820f5482e445fa1120d97d1f1a724
I20260619 18:46:45.474027 26503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: loading cluster ID for follower catalog manager: success
I20260619 18:46:45.477969 26503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: acquiring CA information for follower catalog manager: success
W20260619 18:46:45.479501 26503 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260619 18:46:45.481165 26474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8611ceee130f46e1b358817b2dfc5bd8: Generated new TSK 0
I20260619 18:46:45.482370 26474 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:46:45.591136 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.577996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.3.85.189:35677)
0619 18:46:45.578410 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.578416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.588853 (+ 10437us) server_negotiation.cc:407] Connection header received
0619 18:46:45.589246 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.589255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.589369 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.589447 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.590848 (+ 1401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36826: BlockingRecv error: recv got EOF from 127.0.0.1:36826 (error 108)
Metrics: {"server-negotiator.queue_time_us":291}
W20260619 18:46:45.594134 26441 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.577812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.3.85.190:39271)
0619 18:46:45.578252 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.578259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.592137 (+ 13878us) server_negotiation.cc:407] Connection header received
0619 18:46:45.592478 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.592485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.592596 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.592674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.593872 (+ 1198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36012: BlockingRecv error: recv got EOF from 127.0.0.1:36012 (error 108)
Metrics: {"server-negotiator.queue_time_us":248}
W20260619 18:46:45.595269 26443 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.579737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.3.85.188:41897)
0619 18:46:45.579926 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.579932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.593104 (+ 13172us) server_negotiation.cc:407] Connection header received
0619 18:46:45.593626 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.593632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.593724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.593799 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.594978 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32864: BlockingRecv error: recv got EOF from 127.0.0.1:32864 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 18:46:45.603777 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.601166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.3.85.189:35677)
0619 18:46:45.601365 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.601370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.601934 (+ 564us) server_negotiation.cc:407] Connection header received
0619 18:46:45.602207 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.602214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.602368 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.602546 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.603439 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36828: BlockingRecv error: recv got EOF from 127.0.0.1:36828 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 18:46:45.611896 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.609039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.3.85.189:35677)
0619 18:46:45.609247 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.609254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.609802 (+ 548us) server_negotiation.cc:407] Connection header received
0619 18:46:45.610012 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.610017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.610112 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.610226 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.611653 (+ 1427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36840: BlockingRecv error: recv got EOF from 127.0.0.1:36840 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:46:45.647073 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.644576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.3.85.189:35677)
0619 18:46:45.644796 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.644803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.645358 (+ 555us) server_negotiation.cc:407] Connection header received
0619 18:46:45.645571 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.645577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.645690 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.645818 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.646763 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36846: BlockingRecv error: recv got EOF from 127.0.0.1:36846 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260619 18:46:45.658608 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.655734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.3.85.189:35677)
0619 18:46:45.655997 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.656004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.656593 (+ 589us) server_negotiation.cc:407] Connection header received
0619 18:46:45.657051 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.657059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.657184 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.657310 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.658238 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36858: BlockingRecv error: recv got EOF from 127.0.0.1:36858 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260619 18:46:45.673080 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.670502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.3.85.189:35677)
0619 18:46:45.670759 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.670765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.671295 (+ 530us) server_negotiation.cc:407] Connection header received
0619 18:46:45.671503 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.671511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.671624 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.671766 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.672807 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36874: BlockingRecv error: recv got EOF from 127.0.0.1:36874 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 18:46:45.696256 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.693659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.3.85.189:35677)
0619 18:46:45.693873 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.693879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.694532 (+ 653us) server_negotiation.cc:407] Connection header received
0619 18:46:45.694772 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.694778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.694883 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.695014 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.695972 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36878: BlockingRecv error: recv got EOF from 127.0.0.1:36878 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 18:46:45.739051 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.736299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.3.85.189:35677)
0619 18:46:45.736506 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.736512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.737065 (+ 553us) server_negotiation.cc:407] Connection header received
0619 18:46:45.737350 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.737357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.737469 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.737603 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.738693 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36890: BlockingRecv error: recv got EOF from 127.0.0.1:36890 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 18:46:45.815421 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.812932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.3.85.189:35677)
0619 18:46:45.813141 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.813147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.813713 (+ 566us) server_negotiation.cc:407] Connection header received
0619 18:46:45.813924 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.813937 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.814051 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.814202 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.815135 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36900: BlockingRecv error: recv got EOF from 127.0.0.1:36900 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:46:45.948984 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:45.946975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.3.85.189:35677)
0619 18:46:45.947179 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:45.947184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:45.947550 (+ 366us) server_negotiation.cc:407] Connection header received
0619 18:46:45.947707 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:45.947711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:45.947804 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:45.947903 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:45.948720 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36904: BlockingRecv error: recv got EOF from 127.0.0.1:36904 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:46:46.210855 26367 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:46:46.208810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.3.85.189:35677)
0619 18:46:46.209036 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.209041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.209416 (+ 375us) server_negotiation.cc:407] Connection header received
0619 18:46:46.209602 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.209607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.209693 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.209786 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:46.210655 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36912: BlockingRecv error: recv got EOF from 127.0.0.1:36912 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 18:46:46.410910 26486 catalog_manager.cc:1259] Loaded cluster ID: 21d820f5482e445fa1120d97d1f1a724
I20260619 18:46:46.411063 26486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: loading cluster ID for follower catalog manager: success
I20260619 18:46:46.413987 26486 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: acquiring CA information for follower catalog manager: success
I20260619 18:46:46.416199 26486 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8577dcdb254f4b55b9b8dfa98d4c9aba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:46:46.483300 26503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 165c6c45dfa0452485917505c0a80843: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:46:46.741052 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.732665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.3.85.189:35677)
0619 18:46:46.733302 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.733309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.733787 (+ 478us) server_negotiation.cc:407] Connection header received
0619 18:46:46.733996 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.734001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.734203 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.734320 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:46.735004 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.735879 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.738027 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.738734 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.738905 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.739564 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:46.740263 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:46.740472 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.740563 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.740672 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":278,"threads_started":1}
I20260619 18:46:46.760622 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.748017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.3.85.188:41897)
0619 18:46:46.753339 (+ 5322us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.753344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.753366 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:46:46.753467 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.753471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.753638 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.753819 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:46.754683 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.755796 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.757938 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.758413 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.758600 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.759278 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:46.759865 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:46.759980 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.760116 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.760207 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5191,"thread_start_us":5007,"threads_started":1}
I20260619 18:46:46.779495 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.767347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.3.85.189:35677)
0619 18:46:46.770798 (+ 3451us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.770804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.771231 (+ 427us) server_negotiation.cc:407] Connection header received
0619 18:46:46.771339 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.771348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.771476 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.771585 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:46.772509 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.773882 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.776776 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.777408 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.777531 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.778194 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:46.778334 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:46.778416 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.779088 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.779198 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3292}
I20260619 18:46:46.782403 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.766812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.3.85.190:39271)
0619 18:46:46.774752 (+ 7940us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.774758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.775364 (+ 606us) server_negotiation.cc:407] Connection header received
0619 18:46:46.775495 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.775501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.775757 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.775940 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:46.776773 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.777832 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.780038 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.780659 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.780791 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.781337 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:46.781688 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:46.781768 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.781970 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.782060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7801,"thread_start_us":7675,"threads_started":1}
I20260619 18:46:46.782867 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:46.767385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.3.85.188:41897)
0619 18:46:46.767593 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:46.767602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:46.775881 (+ 8279us) server_negotiation.cc:407] Connection header received
0619 18:46:46.776157 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:46.776176 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:46.776263 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:46.776328 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:46:46.777148 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.778299 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.780783 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:46.781408 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:46.781485 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:46.782052 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:46:46.782190 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:46:46.782285 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:46.782482 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 18:46:46.782572 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:46:46.787689 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 26221
I20260619 18:46:46.788862 26283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:46.906486 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 26221
I20260619 18:46:46.930917 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 26295
I20260619 18:46:46.932528 26357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:47.047537 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 26295
I20260619 18:46:47.070135 3414 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu pid 26370
I20260619 18:46:47.071376 26432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:47.180330 3414 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHiUM6J/build/asan/bin/kudu with pid 26370
2026-06-19T18:46:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5128 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9964 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (377494 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST